Methods and apparatus for use with multiple optical chains

ABSTRACT

Methods and apparatus for supporting zoom operations using a plurality of optical chain modules, e.g., camera modules, are described. Switching between use of groups of optical chains with different focal lengths is used to support zoom operations. Digital zoom is used in some cases to support zoom levels corresponding to levels between the zoom levels of different optical chain groups or discrete focal lengths to which optical chains may be switched. In some embodiments optical chains have adjustable focal lengths and are switched between different focal lengths. In other embodiments optical chains have fixed focal lengths with different optical chain groups corresponding to different fixed focal lengths. Composite images are generate from images captured by multiple optical chains of the same group and/or different groups. Composite image is in accordance with a user zoom control setting. Individual composite images may be generated and/or a video sequence.

RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/246,316 filed Aug. 24, 2016 which is a continuation of U.S.patent application Ser. No. 14/523,918 filed Oct. 26, 2014 which claimsthe benefit of U.S. Provisional Patent Application Ser. No. 61/896,069filed Oct. 26, 2013 and the benefit of U.S. Provisional PatentApplication Ser. No. 62/068,682 filed Oct. 25, 2014 and which is acontinuation-in-part of U.S. patent application Ser. No. 14/327,508filed Jul. 9, 2014 with each of the afore mentioned patent applicationsbeing hereby expressly incorporated by reference in their entirety.

FIELD

The present application relates to zoom methods and apparatus and, moreparticularly, to methods and apparatus for supporting zoom operations byusing multiple optical chains in a camera device.

BACKGROUND

High quality digital cameras have to a large extent replaced filmcameras. However, like film cameras, with digital cameras much attentionhas been placed by the camera industry on the size and quality of lenseswhich are used on the camera. Individuals seeking to take qualityphotographs are often encouraged to invest in large bulky and oftencostly lenses for a variety of reasons. Among the reasons for usinglarge aperture lenses is their ability to capture a large amount oflight in a given time period as compared to smaller aperture lenses.Telephoto lenses tend to be large not only because of their largeapertures but also because of their long focal lengths. Generally, thelonger the focal length, the larger the lens. A long focal length givesthe photographer the ability to take pictures from far away.

In the quest for high quality photos, the amount of light which can becaptured is often important to the final image quality. Having a largeaperture lens allows a large amount of light to be captured allowing forshorter exposure times than would be required to capture the same amountof light using a small lens. The use of short exposure times can reduceblurriness especially with regard to images with motion. The ability tocapture large amounts of light can also facilitate the taking of qualityimages even in low light conditions. In addition, using a large aperturelens makes it possible to have artistic effects such as small depth offield for portrait photography.

While large lenses have many advantages with regard to the ability tocapture relatively large amounts of light compared to smaller lenses,support large zoom ranges, and often allow for good control over focus,there are many disadvantages to using large lenses. Large lenses tend tobe heavy requiring relatively strong and often large support structuresto keep the various lenses of a camera assembly in alignment. The heavyweight of large lenses makes cameras with such lenses difficult andbulky to transport. Furthermore, cameras with large lenses often need atripod or other support to be used for extended periods of time giventhat the sheer weight of a camera with a large lens can become tiresomefor an individual to hold in a short amount of time. In addition toweight and size drawbacks, large lenses also have the disadvantage ofbeing costly.

For a lens of a digital camera to be useful, it needs to be paired witha device which detects the light passing through the lens and convertsit to pixel (picture element) values. A megapixel (MP or Mpx) is onemillion pixels. The term is often used to indicate the number of pixelsin an image or to express the number of image sensor elements of adigital camera where each sensor element normally corresponds to onepixel. Multi-color pixels normally include one pixel value for each ofthe red, green, and blue pixel components.

In digital cameras, the photosensitive electronics used as the lightsensing device is often either a charge coupled device (CCD) or acomplementary metal oxide semiconductor (CMOS) image sensor, comprisinga large number of single sensor elements, each of which records ameasured intensity level. In many digital cameras, the sensor array iscovered with a patterned color filter mosaic having red, green, and blueregions in an arrangement. In such a filter based approach to capturinga color image, each sensor element can record the intensity of a singleprimary color of light. The camera then will normally interpolate thecolor information of neighboring sensor elements, through a processsometimes called demosaicing, to create the final image. The sensorelements in a sensor array using a color filter are often called“pixels”, even though they only record 1 channel (only red, or green, orblue) of the final color image due to the filter used over the sensorelement.

While small focal length lenses paired with relatively high resolutionsensors have achieved widespread commercial success in cell phones andpocket cameras, they often leave their owners longing for better picturequality, e.g., picture quality that can only be achieved with a largerpixel area and a larger lens opening to collect more light.

Smaller sensors require smaller focal length lenses (hence smallerlenses) to frame the same scene from the same point. Availability ofhigh pixel count small sensors means that a smaller lens can be used.However, there are a few disadvantages to using smaller sensors andlenses. First, the small pixel size limits the dynamic range of thesensor as only a small amount of light can saturate the sensor. Second,small lenses collect less total light which can result in grainypictures. Third, small lenses have small maximum apertures which makeartistic effects like small depth of field for portrait pictures notpossible.

In view of the above discussion, it should be appreciated that there isa need for new photographic methods and apparatus which can provide somecombination of the benefits commonly associated with large lenses, e.g.,a relatively large lens area for capturing light, with at least some ofthe benefits of small focal length lenses, e.g., compact size.Additionally, it would be desirable if some of the disadvantages such aslimited dynamic range and/or depth of field associated with small focallength lenses could be avoided and/or such advantages reduced withoutrequiring the use of large lenses.

In particular, there is a need for improved methods and apparatus forsupporting zoom operations. It would be desirable if at least some ofthe methods and/or apparatus provide one or more of the benefitsassociated with use of a large lens. It would be preferable but notabsolutely necessary that the one or more benefits could be providedwithout the need for a large heavy lens extending out way beyond thebody of the camera.

SUMMARY OF THE INVENTION

Methods and apparatus for supporting zoom operations using a pluralityof optical chain modules, e.g., camera modules, are described. Switchingbetween use of groups of optical chains with different focal lengths isused to support zoom operations. Digital zoom is used in some cases tosupport zoom levels corresponding to levels between the zoom levels ofdifferent optical chain groups or discrete focal lengths to whichoptical chains may be switched. In some embodiments optical chains haveadjustable focal lengths and are switched between different focallengths. In other embodiments optical chains have fixed focal lengthswith different optical chain groups corresponding to different fixedfocal lengths. Composite images are generate from images captured bymultiple optical chains of the same group and/or different groups.Composite image is in accordance with a user zoom control setting.Individual composite images may be generated and/or a video sequence.

In some embodiments but not necessary all embodiments, two or moregroups of optical chains each include at least two optical chains whichcapture images in parallel but in some embodiment 3, 4, 5 or moreoptical chains are included in an individual optical chain group. Theimage portions captured by different optical chains in a group may bepartially overlapping, non-overlapping or fully overlapping with variouscombinations and amounts of overlap being a function of the particularimplementation. Groups of optical chains corresponding to differentfocal lengths may include different numbers of optical chains. In oneparticular exemplary embodiment optical chains in a first groupcorresponding to a first focal length include at least 4, and in someembodiments 5, optical chain modules, e.g., with 4 of the modulescapturing substantially non-overlapping portions, e.g., quarters, of ascene area and the fifth capturing the center of the scene area andoverlapping the portions captured by the other optical chain modules ofthe first group. In the particular exemplary embodiment the second groupof optical chains includes a similar configuration to the first, e.g.,with four optical chains being used to capture different portions ofscene area and a fifth to capture the center portion of the scene areain a manner that overlaps the portions captured by the other fouroptical chains of the second group of optical chains. The optical chainsin the second group may, and in some embodiments do, have a focal lengthwhich is smaller than the focal length of the optical chains of thefirst group. A third group of optical chains which may also be used incombination with the first and second groups of optical chains. Thethird group of optical chains, in some embodiments, includes one or morecamera modules having a focal length shorter than the focal length ofthe first and second optical chains. A composite image, in someembodiments is generated using images captured by two or more of thegroups of optical chains with different focal lengths during an imagecapture time period in which the optical chains of the different cameramodules capture an image in parallel. While five optical chains are usedin the first and second groups in other embodiments four optical chainsare used per group with the optical chain used to capture the centerportion of the image being omitted in some embodiments. In otherembodiments the groups of optical chains include 4, 3, 3 or feweroptical chain modules with some groups including a single optical chainmodule while another group may include two or more optical chainmodules. Implementations where at least some groups of optical chainswith different focal lengths include less than 4 optical chains may beparticularly desirable in low cost embodiments which are well suited forcell phones, tablets or other devices where the cost and/or spacerequired for large numbers of optical chains may be an issue. Whilefixed focus length optical chains are used in some embodiments, whetherthe optical chains are fixed focal length or capable of being controlledto change between discrete focal lengths autofocus support may and insome embodiments is provided. However, fixed focus optical chains mayand are used in some embodiments.

Various methods and apparatus of the present invention are directed tomethods and apparatus for obtaining some or all of the benefits of usingrelatively large and long lens assemblies without the need for largelens and/or long lens assemblies, through the use of multiple opticalchain modules in combination.

Optical chain modules including, in some embodiments, relatively shortfocal length lenses which require relatively little depth within acamera are used in some embodiments. While use of short focal lengthlens can have advantages in terms of small lens width, the methods andapparatus of the present are not limited to the use of such lenses andcan be used with a wide variety of lens types. In addition, whilenumerous embodiments are directed to autofocus embodiments, fixed focusembodiments are also possible and supported. An optical chain, invarious embodiments, includes a first lens and an image sensor.Additional lenses and/or one or more optical filters may be includedbetween the first lens of an optical chain module and the image sensordepending on the particular embodiment. In some cases there may be oneor more optical filters before the first lens.

The use of multiple optical chain modules is well suited for use indevices such as cell phones and/or portable camera devices intended tohave a thin form factor, e.g., thin enough to place in a pocket orpurse. By using multiple optical chains and then combining the capturedimages or portions of the captured images to produce a combined image,improved images are produced as compared to the case where a singleoptical chain module of the same size is used.

While in various embodiments separate image sensors are used for each ofthe individual optical chain modules, in some embodiments the imagesensor of an individual optical chain module is a portion of a CCD orother optical sensor dedicated to the individual optical chain modulewith different portions of the same sensor serving as the image sensorsof different optical chain modules.

In various embodiments, images of a scene area are captured by differentoptical chain modules and then subsequently combined either by theprocessor included in the camera device which captured the images or byanother device, e.g., a personal or other computer which processes theimages captured by the multiple optical chains after offloading from thecamera device which captured the images. The combined image has, in someembodiments a dynamic range that is larger than the dynamic range of anindividual image used to generate the combined image.

An exemplary method of operating a camera device including a pluralityof optical chains, said plurality of optical chains including at least afirst group of optical chains and a second group of optical chains, inaccordance with some embodiments, includes: capturing images using thesecond group of optical chains during a first period of time; andcapturing images using the first group of optical chains during a secondperiod of time, optical chains in said first group of optical chainshave a different focal length than optical chains in said second groupof optical chains during at least a portion of said second period oftime. An exemplary camera device, in accordance with some embodiments,includes: a plurality of optical chains, said plurality of opticalchains including at least a first group of optical chains and a secondgroup of optical chains; a first image capture module configured tocapture images using the second group of optical chains during a firstperiod of time; and a second image capture module configured to captureimages using the first group of optical chains during a second period oftime, optical chains in said first group of optical chains have adifferent focal length than optical chains in said second group ofoptical chains during at least a portion of said second period of time.Various described methods and apparatus use multiple groups of lenses tosupport continuous zooming with a combination of digital zoom anddiscrete lens focal length changes.

In some embodiments, but not necessarily all embodiments, the opticalchains do not have lenses which extend out far beyond the body of thecamera and in fact in some embodiments the opening of the optical chainsis flat, flush or nearly flush with the surface of the camera throughwhich light enters an individual optical chain. The covering of anaperture corresponding to an optical chain may be a flat piece of glassor plastic in some embodiments.

Numerous additional features and embodiments are described in thedetailed description which follows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is an exemplary block diagram of an exemplary apparatus, e.g.,camera device, implemented in accordance with one embodiment of thepresent invention.

FIG. 1B illustrates a frontal view of an apparatus implemented inaccordance with an exemplary embodiment of the present invention whichincorporates multiple optical chain modules in accordance with thepresent invention with lenses which are viewable from the front of thecamera.

FIG. 1C, which is a side view of the exemplary apparatus of FIG. 1B,illustrates further details of the exemplary apparatus.

FIG. 1D illustrates a plurality of optical chain modules that can beused in an exemplary device implemented in accordance with theinvention.

FIG. 2 illustrates a camera device implemented in accordance with oneembodiment of the present invention.

FIG. 3A shows an exemplary lens configuration which may be used for theset of outer lenses of the camera device shown in FIGS. 1A-1C.

FIG. 3B illustrates an exemplary filter arrangement which is used in thecamera of FIGS. 1A-1C in some embodiments.

FIG. 3C shows an exemplary inner lens configuration which may, and insome embodiments is, used for a set of inner lenses of the camera deviceshown in FIGS. 1A-1C.

FIG. 4 illustrates an exemplary camera device in which the sets of outerlenses, filters, and inner lenses are mounted on corresponding platters.

FIG. 5A illustrates various filter and lens platters that may be used inthe camera device shown in FIG. 4 depending on the particularembodiment.

FIG. 5B illustrates the filter platter arrangement shown in FIG. 5A whenviewed from the side and when viewed from the front.

FIG. 6A shows lenses, e.g., outer lens plane lenses, corresponding toseven optical chain modules in accordance with some embodiments.

FIG. 6B shows an exemplary color filter arrangement used in someembodiments.

FIG. 6C shows lenses, e.g., inner lens plane lenses, corresponding toseven optical chain modules in accordance with some embodiments.

FIG. 7A shows various lenses, e.g., outer lens plane lenses,corresponding to seven optical chain modules in accordance with someembodiments.

FIG. 7B shows an exemplary color filter arrangement in addition to anindication of exposure time used in some embodiments.

FIG. 7C shows lenses, e.g., inner lens plane lenses, corresponding toseven optical chain modules in accordance with some embodiments.

FIG. 8 illustrates an optical chain arrangement used in one panoramiccamera embodiment in which multiple optical chains and different lensangles are used to capture images that are well suited for combininginto a panoramic image.

FIG. 9 illustrates an exemplary method of producing at least one imageof a first scene area by operating a plurality of optical chain modulesin accordance with one embodiment of the present invention.

FIG. 10 illustrates an exemplary method of producing at least one imageof a first scene area with an enhanced sensor dynamic range by operatingtwo or more optical chain modules in accordance with one embodiment ofthe present invention.

FIG. 11 illustrates an exemplary method of producing at least one imageof a first scene area with enhanced sensor dynamic range by operatingtwo or more optical chain modules in accordance with one embodiment ofthe present invention.

FIG. 12 illustrates an exemplary method of producing at least one imageof a first scene area by operating two or more optical chain modulesusing color filters in accordance with one embodiment of the presentinvention.

FIG. 13 illustrates an exemplary assembly of modules, which may, and insome embodiments is, part of an apparatus which implements one or moremethods of the invention, for performing various image and dataprocessing functions in accordance with one or more exemplaryembodiments of the invention.

FIG. 14 illustrates a computer system which can be used for postprocessing of images captured using a camera device.

FIG. 15 illustrates a frontal view of an apparatus implemented inaccordance with one embodiment of the present invention whichincorporates multiple optical chain modules, e.g., one for each of red,green and blue and one for all three colors.

FIG. 16 illustrates a frontal view of the outer lens assembly of anapparatus implemented in accordance with one embodiment of the presentinvention where the apparatus incorporates multiple optical chainmodules and outer lenses configured with little or no gaps between thelenses.

FIG. 17 illustrates a frontal view of the outer lenses of a lensassembly implemented in accordance with one embodiment of the presentinvention where the apparatus incorporates multiple optical chainmodules with lenses configured with little or no gaps between the lensesbut non-uniform spacing between the optical centers of at least some ofthe lenses.

FIG. 18 illustrates a camera device including a plurality of opticalchain modules which includes mirrors or another device for changing theangle of light entering the optical chain module and thereby allowing atleast a portion of the optical chain module to extend in a direction,e.g., a perpendicular direction, which is not a straight front to backdirection with respect to the camera device.

FIG. 19 illustrates another camera device including a plurality ofoptical chain modules which includes mirrors or another device forchanging the angle of light entering the optical chain module andthereby allowing at least a portion of the optical chain module toextend in a direction, e.g., a perpendicular direction, which is not astraight front to back direction with respect to the camera device.

FIG. 20 illustrates an additional exemplary camera device in whichmirrors and/or other light redirecting elements are used to alter thepath of light in the optical chains so that both the input lenses and/oropenings through which light enters the optical chains can be arrangedin a plane, and also so that the optical sensors of the optical chainscan be arranged in a plane, while allowing at least a portion of thelight path through the optical chains to extend in a directionperpendicular to the input and/or output planes.

FIG. 21 illustrates an additional exemplary camera device in whichmirrors and/or other light redirecting elements are used to alter thepath of light in the optical chains so that the input lenses and/oropenings, as well as the light sensors of the different optical chains,can be arranged in one or more planes at the front of the camera.

FIG. 22A is the first part of a flowchart of an exemplary method ofoperating a camera device including a plurality of optical chains inaccordance with an exemplary embodiment.

FIG. 22B is the second part of a flowchart of an exemplary method ofoperating a camera device including a plurality of optical chains inaccordance with an exemplary embodiment.

FIG. 23 is a drawing illustrating an example in which the method offlowchart is implemented corresponding to a zoom in operation.

FIG. 24 is a drawing illustrating an example in which the method offlowchart is implemented corresponding to a zoom out operation. As theactual and/or simulated focal length decreases an item in the capturedimage will appear to become smaller with a larger area being captured asthe focal length decreases.

FIG. 25A is a first part of an assembly of modules, which may beincluded in an exemplary device, e.g., a camera device including groupsof optical chains, or exemplary combination of devices, e.g., a cameradevice including groups of optical chains and a computer device externalto the camera device, implementing the method of the flowchart of FIG.22.

FIG. 25B is a second part of an assembly of modules, which may beincluded in an exemplary device, e.g., a camera device including groupsof optical chains, or exemplary combination of devices, e.g., a cameradevice including groups of optical chains and a computer device externalto the camera device, implementing the method of the flowchart of FIG.22.

FIG. 26A is a first part of a flowchart of an exemplary method of usingmultiple optical chains including at least a first group of opticalchains and a second group of optical chains in accordance with anexemplary embodiment.

FIG. 26B is a second part of a flowchart of an exemplary method of usingmultiple optical chains including at least a first group of opticalchains and a second group of optical chains in accordance with anexemplary embodiment.

FIG. 27 is an exemplary camera device in accordance with an exemplaryembodiment.

FIG. 28 is a diagram showing how 17 optical chains, e.g., cameramodules, of a camera implemented in accordance with an exemplaryembodiment can be arranged within the body of the camera device.

FIG. 29 is a diagram showing scene areas captured by different opticalchain modules of an exemplary camera device such as the camera device ofFIG. 28.

FIG. 30 shows portions of a scene captured by a camera device such asthe one shown in FIG. 28 and how the scene portions may relate to thescene areas shown in FIG. 29.

DETAILED DESCRIPTION

FIG. 1A illustrates an exemplary apparatus 100, sometimes referred tohereinafter as a camera device, implemented in accordance with oneexemplary embodiment of the present invention. The camera device 100, insome embodiments, is a portable device, e.g., a cell phone or tabletincluding a camera assembly. In other embodiments, it is fixed devicesuch as a wall mounted camera.

FIG. 1A illustrates the camera device 100 in block diagram form showingthe connections between various elements of the apparatus 100. Theexemplary camera device 100 includes a display device 102, an inputdevice 106, memory 108, a processor 110, a transceiver interface 114,e.g., a cellular interface, a WIFI interface, or a USB interface, an I/Ointerface 112, and a bus 116 which are mounted in a housing representedby the rectangular box touched by the line leading to reference number100. The input device 106 may be, and in some embodiments is, e.g.,keypad, touch screen, or similar device that may be used for inputtinginformation, data and/or instructions. The display device 102 may be,and in some embodiments is, a touch screen, used to display images,video, information regarding the configuration of the camera device,and/or status of data processing being performed on the camera device.In the case where the display device 102 is a touch screen, the displaydevice 102 serves as an additional input device and/or as an alternativeto the separate input device, e.g., buttons, 106. The I/O interface 112couples the display 102 and input device 106 to the bus 116 andinterfaces between the display 102, input device 106 and the otherelements of the camera which can communicate and interact via the bus116. In addition to being coupled to the I/O interface 112, the bus 116is coupled to the memory 108, processor 110, an optional autofocuscontroller 132, a transceiver interface 114, and a plurality of opticalchain modules 130, e.g., N optical chain modules. In some embodiments Nis an integer greater than 2, e.g., 3, 4, 7 or a larger value dependingon the particular embodiment. Images captured by individual opticalchain modules in the plurality of optical chain modules 130 can bestored in memory 108, e.g., as part of the data/information 120 andprocessed by the processor 110, e.g., to generate one or more compositeimages. Multiple captured images and/or composite images may beprocessed to form video, e.g., a series of images corresponding to aperiod of time. Transceiver interface 114 couples the internalcomponents of the camera device 100 to an external network, e.g., theInternet, and/or one or more other devices e.g., memory or stand alonecomputer. Via interface 114 the camera device 100 can and does outputdata, e.g., captured images, generated composite images, and/orgenerated video. The output may be to a network or to another externaldevice for processing, storage and/or to be shared. The captured imagedata, generated composite images and/or video can be provided as inputdata to another device for further processing and/or sent for storage,e.g., in external memory, an external device or in a network.

The transceiver interface 114 of the camera device 100 may be, and insome instances is, coupled to a computer so that image data may beprocessed on the external computer. In some embodiments the externalcomputer has a higher computational processing capability than thecamera device 100 which allows for more computationally complex imageprocessing of the image data outputted to occur on the externalcomputer. The transceiver interface 114 also allows data, informationand instructions to be supplied to the camera device 100 from one ormore networks and/or other external devices such as a computer or memoryfor storage and/or processing on the camera device 100. For example,background images may be supplied to the camera device to be combined bythe camera processor 110 with one or more images captured by the cameradevice 100. Instructions and/or data updates can be loaded onto thecamera via interface 114 and stored in memory 108.

The camera device 100 may include, and in some embodiments does include,an autofocus controller 132 and/or autofocus drive assembly 134. Theautofocus controller 132 is present in at least some autofocusembodiments but would be omitted in fixed focus embodiments. Theautofocus controller 132 controls adjustment of at least one lensposition in the optical chain modules used to achieve a desired, e.g.,user indicated, focus. In the case where individual drive assemblies areincluded in each optical chain module, the autofocus controller 132 maydrive the autofocus drive of various optical chain modules to focus onthe same target. As will be discussed further below, in some embodimentslenses for multiple optical chain modules are mounted on a singleplatter which may be moved allowing all the lenses on the platter to bemoved by adjusting the position of the lens platter. In some suchembodiments the autofocus drive assembly 134 is included as an elementthat is external to the individual optical chain modules with the driveassembly 134 driving the platter including the lenses for multipleoptical chains under control of the autofocus controller 132. While theoptical chain modules will in many embodiments be focused together tofocus on an object at a particular distance from the camera device 100,it is possible for different optical chain modules to be focused todifferent distances and in some embodiments different focus points areintentionally used for different optical chains to increase the postprocessing options which are available.

The processor 110 controls operation of the camera device 100 to controlthe elements of the camera device 100 to implement the steps of themethods described herein. The processor may be a dedicated processorthat is preconfigured to implement the methods. However, in manyembodiments the processor 110 operates under direction of softwaremodules and/or routines stored in the memory 108 which includeinstructions that, when executed, cause the processor to control thecamera device 100 to implement one, more or all of the methods describedherein. Memory 108 includes an assembly of modules 118 wherein one ormore modules include one or more software routines, e.g., machineexecutable instructions, for implementing the image capture and/or imagedata processing methods of the present invention. Individual stepsand/or lines of code in the modules of 118 when executed by theprocessor 110 control the processor 110 to perform steps of the methodof the invention. When executed by processor 110, the data processingmodules 118 cause at least some data to be processed by the processor110 in accordance with the method of the present invention. Theresulting data and information (e.g., captured images of a scene,combined images of a scene, etc.) are stored in data memory 120 forfuture use, additional processing, and/or output, e.g., to displaydevice 102 for display or to another device for transmission, processingand/or display. The memory 108 includes different types of memory forexample, Random Access Memory (RAM) in which the assembly of modules 118and data/information 120 may be, and in some embodiments are stored forfuture use. Read only Memory (ROM) in which the assembly of modules 118may be stored for power failures. Non-volatile memory such as flashmemory for storage of data, information and instructions may also beused to implement memory 108. Memory cards may be added to the device toprovide additional memory for storing data (e.g., images and video)and/or instructions such as programming. Accordingly, memory 108 may beimplemented using any of a wide variety of non-transitory computer ormachine readable mediums which serve as storage devices.

Having described the general components of the camera device 100 withreference to FIG. 1A, various features relating to the plurality ofoptical chain modules 130 will now be discussed with reference to FIGS.1B and 1C which show the camera device 100 from front and sideperspectives, respectively. Dashed line 101 of FIG. 1B indicates a crosssection line corresponding to the FIG. 1C view.

Box 117 represents a key and indicates that OCM=optical chain module andeach L1 represents an outermost lens in an optical chain module. Box 119represents a key and indicates that S=sensor, F=filter, L=lens, L1represents an outermost lens in an optical chain module, and L2represents an inner lens in an optical chain module.

FIG. 1B shows the front of the camera device 100. Rays of light 131,which is light toward the front of the camera assembly, shown in FIG. 1Cmay enter the lenses located in the front of the camera housing. Fromthe front of camera device 100, the camera device 100 appears as arelatively flat device with the outer rectangle representing the camerahousing and the square towards the center of the camera representing theportion of the front camera body in which the plurality of optical chainmodules 130 is mounted.

FIG. 1C, which shows a side perspective of camera device 100,illustrates three of the seven optical chain modules (OCM 1 121, OCM 7145, OCM 4 133) of the set of optical chain modules 130, display 102 andprocessor 110. OCM 1 121 includes an outer lens L1 103, a filter 123, aninner lens L2 125, and a sensor 127. OCM 1 121 further includesautofocus drive (AFD) 129 for controlling the position of lens L2 125,and exposure control device (ECD) 131 for controlling sensor 127. TheAFD 129 includes a motor or other drive mechanism which can move thelens (or sensor) to which it is connected. While the AFD 129 is showncoupled, e.g., connected, to the lens L2 125 and thus can move theposition of the lens L2 as part of a focus operation, in otherembodiments the AFD 129 is coupled to the sensor 127 and moves theposition of the sensor 127, e.g., to change the distance between thesensor 127 and the lens 125 as part of a focus operation. OCM 7 145includes an outer lens L1 115, a filter 147, an inner lens L2 149, and asensor 151. OCM 7 145 further includes AFD 153 for controlling theposition of lens L2 149 and ECD 155 for controlling sensor 151.

OCM 4 133 includes an outer lens L1 109, a filter 135, an inner lens L2137, and a sensor 139. The AFD 153 includes a motor or other drivemechanism which can move the lens (or sensor) to which it is connected.While the AFD 153 is shown coupled, e.g., connected, to the lens L2 149and thus can move the position of the lens L2 as part of a focusoperation, in other embodiments the AFD 149 is coupled to the sensor 151and moves the position of the sensor 151, e.g., to change the distancebetween the sensor 151 and the lens 149 as part of a focus operation.OCM 4 133 further includes AFD 141 for controlling the position of lensL2 137 and ECD 143 for controlling sensor 139. The AFD 141 includes amotor or other drive mechanism which can move the lens (or sensor) towhich it is connected. While the AFD 141 is shown coupled, e.g.,connected, to the lens L2 137 and thus can move the position of the lensL2 as part of a focus operation, in other embodiments the AFD 141 iscoupled to the sensor 139 and moves the position of the sensor 139,e.g., to change the distance between the sensor 139 and the lens 137 aspart of a focus operation.

While only three of the OCMs are shown in FIG. 1C it should beappreciated that the other OCMS of the camera device 100 may, and insome embodiments do, have the same or similar structure. FIG. 1C and theoptical chain modules (OCMs), also sometimes referred to as opticalcamera modules, illustrated therein are illustrative of the generalstructure of OCMs used in various embodiments. However, as will bediscussed in detail below, numerous modifications and particularconfigurations are possible. Many of the particular configurations willbe discussed below with use of reference to the optical camera modulesshown in FIG. 1C. While reference to elements of FIG. 1C may be made, itis to be understood that the OCMs in a particular embodiment will beconfigured as described with regard to the particular embodiment. Thus,for example, the filter may be of a particular color. Similarly, inembodiments where the filter is expressly omitted and described as beingomitted or an element which allows all light to pass, while referencemay be made to the OCMs of FIG. 1C, it should be appreciated that thefilter will be omitted in an embodiment where it is indicated to beomitted or of such a nature that it passes a broad spectrum of light topass if the embodiment is indicated to have a broadband filter. As willbe discussed below, the elements of the different OCMs may, but need notbe, mounted on a common support device, e.g., disc or platter, allowinga set of filters, lenses or sensors of the different optical chains tobe moved as a set. While in the OCMs of FIG. 1C mirrors are not shown,as will be discussed below, in at least some embodiments one or moremirrors are added to the OCMs to all light to be directed, e.g., toincrease the length of the optical path or make for a more convenientinternal component configuration. It should be appreciated that each ofthe OCMS 121, 145, 133, shown in FIG. 1C will have their own opticalaxis which corresponds to the path light entering the particular OCMwill follow as it passes from the lens 103, 115, or 109 at the front ofthe optical chain and passes through the OCM to the corresponding sensor127, 151, 139.

While the processor 110 is not shown being coupled to the AFD, ECD andsensors 127, 151, 139 it is to be appreciated that such connectionsexist and are omitted from FIG. 1C to facilitate the illustration of theconfiguration of the exemplary OCMs. As should be appreciated the numberand arrangement of lens, filters and/or mirrors can vary depending onthe particular embodiment and the arrangement shown in FIG. 1C isintended to be exemplary and to facilitate an understanding of theinvention rather than limiting in nature.

The front of the plurality of optical chain modules 130 is visible inFIG. 1B with the outermost lens of each optical chain module appearingas a circle represented using a solid line (OCM 1 L1 103, OCM 2 L1 105,OCM 3 L1 107, OCM 4 L1 109, OCM 5 L1 111, OCM 6 L1 113, OCM 7 L1 115).In the FIG. 1B example, the plurality of optical chain modules 130include seven optical chain modules, OCM 1 121, OCM 2 157, OCM 3 159,OCM 4 133, OCM 5 171, OCM 6 173, OCM 7 145, which include lenses (OCM 1L1 103, OCM 2 L1 105, OCM 3 L1 107, OCM 4 L1 109, OCM 5 L1 111, OCM 6 L1113, OCM 7 L1 115), respectively, represented by the solid circles shownin FIG. 1B. The lenses of the optical chain modules are arranged to forma pattern which is generally circular in the FIG. 1B example when viewedas a unit from the front. While a circular arrangement is preferred insome embodiments, non-circular arrangements are used and preferred inother embodiments. In some embodiments while the overall pattern isgenerally or roughly circular, different distances to the center of thegeneral circle and/or different distances from one lens to another isintentionally used to facilitate generation of a depth map and blockprocessing of images which may include periodic structures such asrepeating patterns without the need to identify edges of the repeatingpattern. Such repeating patterns may be found in a grill or a screen.

Note that the individual outer lenses, in combination, occupy an areathat might otherwise have been occupied by a single large lens. Thus,the overall total light capture area corresponding to the multiplelenses of the plurality of chain modules OCM 1 to OCM 7, also sometimesreferred to as optical camera modules, approximates that of a lenshaving a much larger opening but without requiring a single lens havingthe thickness which would normally be necessitated by the curvature of asingle lens occupying the area which the lenses shown in FIG. 1B occupy.

While gaps are shown between the lens openings of the optical chainmodules OCM 1 to OCM 7, it should be appreciated that the lenses may bemade, and in some embodiments are, made so that they closely fittogether minimizing gaps between the lenses represented by the circlesformed by solid lines. While seven optical chain modules are shown inFIG. 1B, it should be appreciated that other numbers of optical chainmodules are possible.

As will be discussed below, the use of seven optical chain modulesprovides a wide degree of flexibility in terms of the types of filtercombinations and exposure times that can be used for different colorswhile still providing an optical camera module that can be used toprovide an image for purposes of user preview of the image area andselection of a desired focal distance, e.g., by selecting an object inthe preview image which is to be the object where the camera modules areto be focused.

For example, in some embodiments, such as the FIG. 6 embodiment, atleast some of the different optical chain modules include filterscorresponding to a single color thereby allowing capture of a singlecolor at the full resolution of the image sensor, e.g., the sensor doesnot include a Bayer filter. In one embodiment two optical chain modulesare dedicated to capturing red light, two optical chain modules arededicated to capturing green light and two optical chain modules arededicated to capturing blue light. The center optical chain module mayinclude a RGB filter or opening which passes all colors with differentportions of the sensor of the center optical chain module being coveredby different color filters, e.g., a Bayer pattern with the optical chainmodule being used to capture all three colors making it easy to generatecolor preview images without having to process the output of multipleoptical chain modules to generate a preview image.

The use of multiple optical chains such as shown in the FIG. 1A-1Cembodiment has several advantages over the use of a single opticalchain. Using multiple optical chains allows for noise averaging. Forexample, given the small sensor size there is a random probability thatone optical chain may detect a different number, e.g., one or more,photons than another optical chain. This may represent noise as opposedto actual human perceivable variations in the image being sensed. Byaveraging the sensed pixel values corresponding to a portion of animage, sensed by different optical chains, the random noise may beaveraged resulting in a more accurate and pleasing representation of animage or scene than if the output of a single optical chain was used.

As should be appreciated, different wavelengths of light will be bent bydifferent amounts by the same lens. This is because the refractive indexof glass (or plastic) which the lens is made of changes with wavelength.Dedication of individual optical chains to a particular color allows forthe lenses for those optical chains to be designed taking intoconsideration the refractive index of the specific range of wavelengthfor that color of light. This can reduce chromatic aberration andsimplify lens design. Having multiple optical chains per color also hasthe advantage of allowing for different exposure times for differentoptical chains corresponding to a different color. Thus, as will bediscussed further below, a greater dynamic range in terms of lightintensity can be covered by having different optical chains usedifferent exposure times and then combining the result to form thecomposite image, e.g., by weighting the pixel values output by thesensors of different optical chains as a function of exposure time whencombining the sensed pixel values to generate a composite pixel valuefor use in a composite image. Given the small size of the opticalsensors (pixels) the dynamic range, in terms of light sensitivity, islimited with the sensors becoming easily saturated under brightconditions. By using multiple optical chains corresponding to differentexposure times the dark areas can be sensed by the sensor correspondingto the longer exposure time while the light areas of a scene can besensed by the optical chain with the shorter exposure time withoutgetting saturated. Pixel sensors of the optical chains that becomesaturated as indicated by a pixel value indicative of sensor saturationcan be ignored, and the pixel value from the other, e.g., less exposed,optical chain can be used without contribution from the saturated pixelsensor of the other optical chain. Weighting and combining ofnon-saturated pixel values as a function of exposure time is used insome embodiments. By combining the output of sensors with differentexposure times a greater dynamic range can be covered than would bepossible using a single sensor and exposure time.

FIG. 1C is a cross section perspective of the camera device 100 shown inFIGS. 1A and 1B. Dashed line 101 in FIG. 1B shows the location withinthe camera device to which the cross section of FIG. 1C corresponds.From the side cross section, the components of the first, seventh andfourth optical chains are visible. As illustrated in FIG. 1C despiteincluding multiple optical chains the camera device 100 can beimplemented as a relatively thin device, e.g., a device less than 2, 3or 4 centimeters in thickness in at least some embodiments. Thickerdevices are also possible, for example devices with telephoto lenses andare within the scope of the invention, but the thinner versions areparticularly well suited for cell phones and/or tablet implementations.

As illustrated in the FIG. 1C diagram, the display device 102 may beplaced behind the plurality of optical chain modules 130 with theprocessor 110, memory and other components being positioned, at least insome embodiments, above or below the display and/or optical chainmodules 130. As will be discussed below, and as shown in FIG. 1C, eachof the optical chains OCM 1 121, OCM 7 145, OCM 4 133 may, and in someembodiments do, include an outer lens L1, an optional filter F, and asecond optional lens L2 which proceed a sensor S which captures andmeasures the intensity of light which passes through the lens L1, filterF and second lens L2 to reach the sensor S. The filter may be a colorfilter or one of a variety of other types of light filters.

In FIG. 1C, each optical chain module includes an auto focus drive (AFD)also sometimes referred to as an auto focus device which can alter theposition of the second lens L2, e.g., move it forward or back, as partof a focus operation. An exposure control device (ECD) which controlsthe light exposure time of the sensor to which the ECD corresponds, isalso included in each of the OCMs shown in the FIG. 1C embodiment. TheAFD of each optical chain module operates under the control of theautofocus controller 132 which is responsive to user input whichidentifies the focus distance, e.g., by the user highlighting an objectin a preview image to which the focus is to be set. The autofocuscontroller while shown as a separate element of the device 100 can beimplemented as a module stored in memory and executed by processor 110.

Note that while supporting a relatively large light capture area andoffering a large amount of flexibility in terms of color filtering andexposure time, the camera device 100 shown in FIG. 1C is relatively thinwith a thickness that is much less, e.g., ⅕th, 1/10th, 1/20th or evenless than the overall side to side length or even top to bottom lengthof the camera device visible in FIG. 1B.

FIG. 1D illustrates a plurality of optical chain modules 160 that can beused in an exemplary device implemented in accordance with theinvention. The optical chain modules (OCMs) shown in FIG. 1D areillustrative of the general structure of OCMs used in variousembodiments. However, as will be discussed in detail below, numerousmodifications and particular configurations are possible. Many of theparticular configurations will be discussed below with use of referenceto the optical camera modules shown in FIG. 1D to support the particularexemplary embodiments. While reference to elements of FIG. 1D may andwill be made with regard to particular embodiments, it is to beunderstood that the OCMs in a particular embodiment will be configuredas described with regard to the particular embodiment. Thus, forexample, in a particular embodiment one or of the OCMS may use filtersof a particular color or may even omit the filter 164, 164′. 164″ or164′″. Similarly, in embodiments where the filter is expressly omittedand described as being omitted or an element which allows all light topass, while reference may be made to the OCMs of FIG. 1D, it should beappreciated that the filter will be omitted in such an embodiment whereit is expressly indicated to be omitted or of such a nature that itpasses a broad spectrum of light to pass if the embodiment is indicatedto have a broadband filter. As will be discussed below, the elements ofthe different OCMs may, but need not be, mounted on a common supportdevice, e.g., disc or platter, allowing a set of filters, lenses orsensors of the different optical chains to be moved as a set. While inthe OCMs of FIG. 1D mirrors are not shown, as will be discussed below,in at least some embodiments one or more mirrors are added to the OCMsto all light to be directed, e.g., to increase the length of the opticalpath or make for a more convenient internal component configuration. Itshould be appreciated that each of the OCMS 164, 164′, 164″. 164′″,shown in FIG. 1C will have their own optical axis which corresponds tothe path light entering the particular OCM will follow as it passes fromthe lens 162, 162′. 162″, 162′″ at the front of the optical chain andpasses through the OCM to the corresponding sensor 168, 168′, 168″,168′″.

The plurality of optical chain modules 160 includes N exemplary opticalchain modules as illustrated in FIG. 1D where N may be any number butusually a number greater than one, and in many cases greater than 2, 6or even a larger number. The plurality of optical chain modules 160includes a first optical chain module (OCM) 161, a second optical chainmodule 161′, a third optical chain module 161″, . . . , and Nth opticalchain module 161′″.

Each optical chain module illustrated in FIG. 1D includes many or all ofthe same elements shown in each optical chain illustrated in FIG. 1Csuch as, e.g., optical chain module 121. The first exemplary OCM 161includes an outer lens 162, a filter 164, an inner lens 166, a sensor168, an auto focus drive (AFD) 169 and an exposure control device (ECD)170. Each of the other optical chain modules include similar elements asdescribed above with regard to the first OCM 160, with the like elementsin each of the other optical chain modules being identified using aprime (′), double prime (″), or triple prime (′″). For example, theexemplary second OCM 161′ includes an outer lens 162′, a filter 164′, aninner lens 166′, a sensor 168′, an auto focus drive (AFD) 169′ and anexposure control device (ECD) 170′, the exemplary third OCM 161″includes an outer lens 162″, a filter 164″, an inner lens 166″, a sensor168″, an auto focus drive (AFD) 169″ and an exposure control device(ECD) 170″ and so on. Similarly, the Nth OCM 161′ includes an outer lens162′″, a filter 164′″, an inner lens 166′″, a sensor 168′″, an autofocus drive (AFD) 169′″ and an exposure control device (ECD) 170′″. Theoperation and functionality of each of the OCMs and their elements isthe same as or similar the functionality of optical chain modulesdiscussed earlier with respect to FIG. 1C and thus will not be repeated.Note that two versions of the AFD 169, 169′, 169″ or 169′″ are shown ineach optical chain module with the AFD connected to a lens being shownusing solid lines and an alternative AFD shown using dashed lines beingconnected to the sensor 168, 168′, 168″ or 168′″. The AFD shown withdashed lines adjusts the position of the sensor 168. 168′, 168″ or 168′″to which it is connected as part of an autofocus operation, e.g., movingthe sensor forward or backward to alter distance between the sensor anda lens. The AFD shown in solid lines is used in systems where a lensrather than a sensor is moved as part of an AFD operation. In someembodiments the AFD controls the position of a lens and/or sensor inwhich case the AFD module is connected to both a lens support mechanismor lens and the sensor.

The plurality of optical chain modules 160 of FIG. 1D can be used as,e.g., the plurality of optical modules 130 of the exemplary device 100or any other device implemented in accordance with the invention. Thenumber and particular configuration of optical chains in the step ofoptical chains 160 maybe as per various embodiments which will bedescribed in the following detailed description. Accordingly, while aparticular embodiment may be described in one more subsequent portionsof this application, it is to be understood that the optical chains 160may be used in such embodiments with the particular configuration offilters, lens, and element supports being as described with respect tothe particular exemplary embodiment being discussed.

FIG. 2 illustrates a camera device 200 implemented in accordance withthe invention. The FIG. 2 camera device 200 includes many or all of thesame elements shown in the device 100 of FIGS. 1A-1C. Exemplary cameradevice 200 includes a plurality of optical chain modules (OCM 1 205, OCM2 207, . . . , OCM N 209, a processor 211, memory 213 and a display 215,coupled together. OCM 1 205 includes outer lens L1 251, filter 253,inner lens L2 255, sensor 1 257, AFD 259 and ECD 261. In someembodiments, processor 211 of camera device 200 of FIG. 2 is the same asprocessor 110 of device 100 of FIG. 1A, memory 213 of device 200 of FIG.2 is the same as memory 108 of device 100 of FIG. 1A, and display 215 ofdevice 200 of FIG. 2 is the same as display 102 of device 100 of FIG.1A.

OCM 2 207 includes outer lens L1 263, filter 265, inner lens L2 267,sensor 2 269, AFD 271 and ECD 273. OCM N 209 includes outer lens L1 275,filter 277, inner lens L2 279, sensor N 281, AFD 283 and ECD 285. Box217, which represents a key, indicates that ECD=exposure control deviceand AFD=auto focus drive.

In the FIG. 2 embodiment the optical chain modules (optical chain module1 205, optical chain module 2 207, . . . , optical chain module N 209)are shown as independent assemblies with the autofocus drive of eachmodule being a separate AFD element (AFD 259, AFD 271, AFD 283),respectively.

In FIG. 2, the structural relationship between the various lenses andfilters which precede the sensor in each optical chain module can beseen more clearly. While three elements, e.g. two lenses (see columns201 and 203 corresponding to L1 and L2, respectively) and the filter(corresponding to column 202) are shown in FIG. 2 before each sensor, itshould be appreciated that a much larger combination of lenses and/orfilters may precede the sensor of one or more optical chain modules withanywhere from 2-10 elements being common and an even larger number ofelements being used in some embodiments, e.g., high end embodimentsand/or embodiments supporting a large number of filter and/or lensoptions.

In some but not all embodiments, optical chain modules are mounted inthe camera device to extend from the front of the camera device towardsthe back, e.g., with multiple optical chain modules being arranged inparallel. Filters and/or lenses corresponding to different optical chainmodules may, and in some embodiments are, arranged in planes extendingperpendicular to the front to back direction of the camera device fromthe bottom of the camera device towards the top of the camera device.While such a mounting arrangement is used in some embodiments, otherarrangements where the optical chain modules are arranged at differentangles to one another and/or the camera body are possible.

Note that the lenses/filters are arranged in planes or columns in thevertical dimension of the camera device 200 to which reference numbers201, 202, 203 correspond. The fact that the lenses/filters are alignedalong vertical planes allows for a manufacturing and structuralsimplification that is used in some embodiments. That is, in someembodiments, the lenses and/or filters corresponding to a plane 201,202, 203 are formed or mounted on a platter or plate. The term platterwill be used for discussion purposes but is not intended to be limiting.The platter may take the form of a disc but non-round platters are alsocontemplated and are well suited for some embodiments. In the case ofplastic lenses, the lenses and platter may be molded out of the samematerial in a single molding operation greatly reducing costs ascompared to the need to manufacture and mount separate lenses. As willbe discussed further, platter based embodiments allow for relativelysimple synchronized focus operations in that a platter may be movedfront or back to focus multiple OCMs at the same time. In addition, aswill be explained, platters may be moved or rotated, e.g., along acentral or non-central axis, to change lenses and or filterscorresponding to multiple optical chain modules in a single operation. Asingle platter may include a combination of lenses and/or filtersallowing, e.g., a lens to be replaced with a filter, a filter to bereplaced with a lens, a filter or lens to be replaced with anunobstructed opening. As should be appreciated the platter basedapproach to lens, filter and/or holes allows for a wide range ofpossible combinations and changes to be made by simple movement of oneor more platters. It should also be appreciated that multiple elementsmay be combined and mounted together on a platter. For example, multiplelenses, filters and/or lens-filter combinations can be assembled andmounted to a platter, e.g., one assembly per optical chain module. Theassemblies mounted on the platter for different optical chains may bemoved together, e.g., by rotating the platter, moving the platterhorizontally or vertically or by moving the platter using somecombination of one or more such movements.

While platters have been described as being moved to change elements inan optical chain, they can, and in some embodiments are, moved for imagestabilization purposes. For example, a platter having one or more lensesmounted thereon can be moved as part of an image stabilizationoperation, e.g., to compensate for camera motion.

While mounting of lenses and filters on platters has been discussed, itshould also be appreciated that the sensors of multiple optical chainscan be mounted on a platter. For example, sensors without color filtersmay be replaced with sensors with color filters, e.g., Bayer patternfilters. In such an embodiment sensors can be swapped or changed whileleaving one or more components of one or more optical chains in place.

Note from a review of FIG. 2 that in some embodiments, e.g., largerfocal length telephoto applications, the elements, e.g., filters/lensescloser to the sensor of the optical chain module, are smaller in sizethan the outer most lenses shown in column 201. As a result of theshrinking size of the lenses/filters, space becomes available betweenthe lenses/filters within the corresponding platter.

FIGS. 3A through 3C provide perspective views of the different planes201, 202, 203 shown in FIG. 2. As shown in FIG. 3A, the outer lenses L1(OCM 1 L1 251, OCM 2 L1 263, OCM 3 L1 264, OCM 4 L1 266, OCM 5 L1 268,OCM 6 L1 270, OCM 7 L1 272) occupy much of the outer circular areacorresponding to the front of the camera modules as previously shown inFIG. 1B. However, as shown in FIG. 3B the filters (OCM 1 F 253, OCM 2 F265, OCM 3 F 274, OCM 4 F 276, OCM 5 F 278, OCM 6 F 280, OCM 7 F 282)corresponding to plane 202 occupy less space than the lenses shown inFIG. 3A while the inner lenses L2 (OCM 1 L2 255, OCM 2 L2 267, OCM 3 L2284, OCM 4 L2 286, OCM 5 L2 288, OCM 6 L2 290, OCM 7 L2 292) shown inFIG. 3C occupy even less space. In some embodiments, where N=7, outerlens L1 275, filter F 277, and inner lens L2 279 of FIG. 2 are the sameas OCM 7 L1 272 of FIG. 3A, OCM 7 F 282 of FIG. 3B and OCM 7 L2 292 ofFIG. 3C, respectively.

The decreasing size of the inner components allow multiple lenses and/orfilters to be incorporated into a platter corresponding to one or moreof the inner planes. Consider for example that an alternative filter F′or hole could be mounted/drilled below or next two each filter F of aplatter corresponding to plan 202 and that by shifting the position orplatter vertically, horizontally or a combination of horizontally andvertically, the filter F can be easily and simply replaced with anotherfilter or hole. Similarly the lenses L2 may be replaced by alternativelenses L2′ by shifting a platter of lenses corresponding to plane 203.In some embodiments, the platter may also be rotated to support changes.The rotation may be an off center rotation and/or may be performed incombination with one or more other platter position changes.

A camera device 60 which includes platters of lenses and/or filters (61,62, 63) is shown in FIG. 4. Camera device 60 includes a plurality ofoptical chain modules (optical chain module 1 69, optical chain module 270, . . . , optical chain module N 71), processor 72, memory 73, anddisplay 74 coupled together via bus 75. In some embodiments, processor72, memory 73, display 74, and autofocus controller 76 of device 60 ofFIG. 4 are the same as processor 110, memory 108, display 102, andautofocus controller 132 of device 100 of FIG. 1A.

Element 61 represents a platter of outer lenses L1 with 3 of the lenses(77, 81, 86) being shown as in the FIG. 1C example. Additional lensesmay be, and often are, included on the platter 61 in addition to theones shown. For example, in a seven optical chain module embodiment suchas shown in FIG. 1, platter 61 would include seven outer lenses. Notethat the thickness of the platter 61 need not exceed the maximumthicknesses of the lenses and from a side perspective is much thinnerthan if a single lens having a similar curvature to that of theindividual lenses L1, but with the single lens being larger, occupiedthe same area as all the 7 lenses on the platter 61. Platter 62 includesthe filters F, which include the three filters (77, 82, 87) whileplatter 63 includes the inner lenses L2, which include the three lenses(78, 83, 88). As can be appreciated the camera device 60 is the same asor similar to the camera device of FIG. 1C and FIG. 2 but with thelenses and filters being mounted on platters which may be moved betweenthe front and back of the camera to support autofocus or horizontallyand/or vertically to support lens/filter changes.

Auto focus drive 66 is used to move platter 63 forward or backward aspart of a focus operation, e.g., under control of the autofocuscontroller 76 which may be, and often is, included in the camera device60. A filter shift drive (FSD) 65 is included in embodiments whereshifting of the platter 62 is supported as part of a filter changeoperation. The FSD 65 is responsive to the processor 72 which operatesin response to user selection of a particular mode of operation and/oran automatically selected mode of operation and can move the platter 62vertically, horizontally or in some combination of vertical andhorizontal motion to implement a filter change operation. The FSD 62 maybe implemented with a motor and mechanical linkage to the platter 62. Insome embodiments, the platter 62 may also be rotated to support changes.The rotation may be an off center rotation and/or may be performed incombination with one or more other platter position changes.

A lens shift drive (LSD) 67 is included in embodiments where shifting ofthe platter 63 is supported as part of a filter change operation. TheLSD 67 is responsive to the processor 72 which operates in response touser selection of a particular mode of operation and/or an automaticallyselected mode of operation and can move the platter 63 vertically,horizontally or in some combination of vertical and horizontal motion toimplement a lens change operation. The LSD 67 may be implemented with amotor and mechanical linkage to the platter 63. In some embodiments, theplatter 63 may also be rotated to support changes. The rotation may bean off center rotation and/or may be performed in combination with oneor more other platter position changes.

FIG. 5A illustrates various exemplary platters that can, and in someembodiments are, used as the filter platter and/or inner lens platter inthe camera device 60 of FIG. 4. In the FIG. 5A example N is three (3)but other values of N are possible depending on the embodiment. FIG. 5Bshows the exemplary lens platter 62′ of FIG. 5A when viewed from theside, drawing 6299, and from the front, drawing 6298.

Platter 62 represents a platter with a single set of filters F1,1 6202corresponding to OCM1, F1,2 6204 corresponding to OCM 2 and F1,3 6206corresponding to OCM 3.

Platter 62′ represents an alternative platter that can, and in someembodiments is, used in place of platter 62. NF is use to represent ahole or No Filter (NF) area of the platter 62′. As should be appreciatedby simply shifting platter 62′ vertically the filters F1 (F1,1 6202,F1,2 6204, F1, 3 6206) can be replaced by holes (NF 6208, NF 6210, NF6212), respectively, thereby removing the color or other types offilters previously included in the optical chain modules.

Platter 62″ of FIG. 5A represents a platter which includes alternativefilters F1′ (F1′, 1 6214, F1′, 2 6216, F1′ 3 6206) which can be switchedfor the filters F1 (F1, 1 6202, F1,2 6204, F1,3 6206), respectively, bymoving the platter 62″ vertically. Thus platter 62″ is used to show howfilters can be switched for other filters by simple movement of aplatter while platter 62′ shows how filters can be removed from theoptical paths included in a plurality of optical chain modules byshifting of the platter on which a set of filters are mounted.

With regard to drawing 6298 of FIG. 5B, as should be appreciated bysimply shifting platter 62′ vertically the filters F1 (F1,1 6202, F1,26204, F1, 3 6206, F1,4 6220, F1, 5 6224, F1, 6 6228, F1, 7 6232) can bereplaced by holes (NF 6208, NF 6210, NF 6212, NF 6222, NF 6226, NF 6230,NF 6234), respectively, thereby removing the color or other types offilters previously included in the optical chain modules.

Lens platter 63 shows a platter of inner lenses L2 (L2,1 6302, L2,26304, L2,3 6306) corresponding to first, second and third optical cameramodules. Lens platter 63′ is an alternative platter which shows howalternative lenses L2′ (L2′,1 6308, L2′,2 6310, L2′,3 6312) can beincluded on a lens platter and easily swapped for the lenses L2 (L2,16302, L2,2 6304, L2,3 6306), respectively, by simple movement of theplatter 63′ vertically or horizontally. Lens platter 63″ is used to showthat a lens platter may include holes (6314, 6316, 6318) as analternative to alternative lenses. Any of lens platters 63, 63′ or 63″could be used in the camera device 60 shown in FIG. 4. While two lenssets are included in platter 63′, multiple lens and/or holecombinations, e.g., 2, 3 or more, may be included in a single platter.Similarly a large number of alternative filter, hole alternatives may besupported in a single filter platter. A platter can also havecombinations of lenses, filters and holes and filters could be swappedfor lenses or holes.

As should be appreciated given the larger number of lens/filtercombinations that can be supported through the use of platters, a singlecamera device including a number of optical chain modules may support alarge number of alternative modes of operation.

It should be appreciated that the exposure control of various opticalchain modules may be varied along with the filters and/or lenses used atany given point in time allowing for a wide degree of flexibility andcontrol over the images captured at any given point in time.

FIGS. 6A, 6B and 6C correspond to one particular filter lens combinationused in some embodiments. FIG. 6, which comprises the combination ofFIGS. 6A, 6B, and 6C, shows an exemplary combination of lenses andfilters used in one exemplary embodiment in which a single color filteris used in at least some of the different optical chain modules.

FIG. 6A shows the use of 7 optical chain modules at plane 201 (the outerlens plane corresponding to lenses L1) as viewed from the front of thecamera device. FIG. 6A shows optical chain module L1 lenses (OCM 1 L16402, OCM 2 L1 6404, OCM 3 L1 6406, OCM 4 L1 6408, OCM 5 L1 6410, OCM 6L1 6412, OCM 7 L1 6414). FIG. 6C shows the inner lens plane 203. FIG. 6Cshows optical chain module L2 lenses (OCM 1 L2 6602, OCM 2 L2 6604, OCM3 L2 6606, OCM 4 L2 6608, OCM 5 L2 6610, OCM 6 L2 6612, OCM 7 L2 6614).The configuration shown in FIGS. 6A and 6C is the same or similar tothat previously discussed with reference to the FIG. 3 embodiment. FIG.6B shows a particular color filter arrangement used in some embodiments.The filter arrangement shown in FIG. 6B may be used at filter plane 202.The filter arrangement shown in FIG. 6B may be used in the set ofoptical chain modules 130 before the sensors, e.g., between the set ofL1 and L2 lenses. However, this position is not required for someembodiments and the user of inner lenses L2 is also not required forsome embodiments.

The filter configuration 6002 of FIG. 6B includes single color filtersin each of a plurality of optical chain modules, e.g., the six outeroptical chain modules (OCM1 to OCM6). Multiple optical chain modules arededicated to each of the three colors, red (R), green (G) and blue (B).The optical chain modules (OCM1, OCM4) with the red filter (RF), (OCM 1RF 6502, OCM 4 RF 6508) pass and sense red light. The optical chainmodules (OCM 2, OCM 5) with the green filter (GF), OCM 2 GF 6504, OCM 5GF 6510, pass and sense green light. The optical chain modules (OCM 3,OCM 6) with the blue filter (BF), OCM 3 BF 6506, OCM 6 BF 6512, pass andsense blue light. In various embodiments, there is a single color filterper lens for the outer lenses, e.g., a single color filter correspondingto each of OCM 1-OCM 6. In some such embodiments, there are multipleOCMs per single color, e.g., 2 OCMs for each of Red, Green, and Blue.

By using optical chain modules dedicated to a single color, the opticalchains can be optimized for the spectral range corresponding to theparticular color to which the chain corresponds. In addition postcapture color compensation can be simplified since each of the six outeroptical modules capture a single known color. In addition, noise can beaveraged between the sensor corresponding to the same color and/ordifferent exposure times can be used for the different OCMscorresponding to an individual color extending the dynamic range of thesensors to cover a range wider than could be captured by a singlesensor. In addition different exposure times may be used for differentcolors to take into consideration particular color biased lightingconditions and/or facilitate the implementation of particular coloreffects that may be desired. Notably the individual colors are capturedat a pixel result in a resolution equal to that of the sensor as opposedto the case where different portions of a single sensor are used tocapture different colors, e.g., with each color R, G, B being capturedat a resolution ⅓ that of the pixel resolution of the image sensor beingused in an optical chain module.

In some embodiments, there is a RGB Multicolor Filter, OCM 7 RGBF 6514,corresponding to OCM 7. In some embodiments, OCM 7 filter 6514 is a RGBfilter, e.g., a Bayer filter. In some embodiments, an opening whichallows all colors to pass is used in place of OCM 7 RGB filter 6514, butthe sensor area corresponding to OCM 7 includes R, G, and B filterscorresponding to different sensor area portions. In some embodiments,OCM 7 is used for preview. In various embodiments, the sensors for OCM 1through OCM 6 have no filters.

While in some embodiments a composite image is generated and displayedas a preview image, in some embodiments to reduce processing time and/orthe time required to display a composite image which may be delayed bythe time required to combine multiple images, an image captured by asingle sensor is displayed as the preview image on the display of thecamera device. The multi-colored filter incorporated into the sensor,e.g., Bayer filter, of OCM 7 allows a color image to be captured by asingle lens and used as the preview image. While the image may be oflower quality than that which can be generated by creating a compositeof the multiple OCMs given the small display size the difference inimage quality between the preview image generated from OCM 7 and that ofa composite image may not be sufficient to justify the processing,power, and/or time required to generate a composite image for previewpurpose. Accordingly, the FIG. 6B filter arrangement provides a greatdeal of flexibility while being able to support a wide variety ofexposure and other image capture related features.

Box 6003 of FIG. 6B identifies that GF=green filter, BF=blue filter,RF=red filter, RGBF=Red, Green, Blue Filter. The ability to usedifferent exposure times with different optical chain modules isillustrated further with regard to a camera embodiment which will now bediscussed with regard to FIGS. 7A, 7B and 7C. FIG. 7, which comprisesthe combination of FIGS. 7A, 7B, and 7C, shows an exemplary combinationof lenses and filters used in one exemplary embodiment in whichexposures of different duration are used for different optical chainmodules and a single color filter is used in at least some of thedifferent optical chain modules. The lens configurations of FIGS. 7A and7C are similar to that shown in FIGS. 6A and 6C. FIG. 7A shows opticalchain module L1 lenses (OCM 1 L1 7402, OCM 2 L1 7404, OCM 3 L1 7406, OCM4 L1 7408, OCM 5 L1 7410, OCM 6 L1 7412, OCM 7 L1 7414), which may belocated a plane 201. FIG. 6C shows optical chain module L2 lenses (OCM 1L2 7602, OCM 2 L2 7604, OCM 3 L2 7606, OCM 4 L2 7608, OCM 5 L2 7610, OCM6 L2 7612, OCM 7 L2 7614), which may be located at plane 203. The filterarrangement shown in drawing 7002 of FIG. 7B is also the same or similarto that shown in FIG. 6B but in the FIG. 7B example exposure time isalso included. While the exposure is controlled by use of the exposurecontrol device in some embodiments the concept can be understood fromFIG. 7B. In FIG. 7B SE is used to indicate short exposure, LE is used toindicate long exposure, and ME is used to indicate medium exposure, asindicated by box 7003. Element 7502 indicates that OCM 1 uses a redfilter and is controlled for a medium exposure. Element 7504 indicatesthat OCM 2 uses a green filter and is controlled for a short exposure.Element 7506 indicates that OCM 3 uses a blue filter and is controlledfor a short exposure. Element 7508 indicates that OCM 4 uses a redfilter and is controlled for a long exposure. Element 7510 indicatesthat OCM 5 uses a green filter and is controlled for a long exposure.Element 7512 indicates that OCM 6 uses a blue filter and is controlledfor a long exposure. Element 7514 indicates that OCM 7 uses a RGBfilter, e.g., a Bayer filter, and is controlled for medium exposure. Forthe outer OCMs, OCM 1 through OCM 6, there is a single color filter perOCM, and multiple OCMs per color. In various embodiments, the centerOCM, OCM 7, is used for preview.

In some embodiments, filters, corresponding to OCM 1 through OCM 7, areincluded at plane 202. In some embodiments, the filters corresponding toOCM 1 through OCM 6 are included at plane 202; there is an opening atplane 2 corresponding to OCM 7, which allows all the colors to pass; andthe sensor area corresponding to OCM 7 includes R, G, and B filterscorresponding to different sensor area portions, e.g., the sensor forOCM 7 includes an RGB Bayer filter. In some embodiments, the sensors forOCM 1 through OCM 6 have no filters.

The preview image is generated using the medium exposure optical chainmodule while the two different optical chain modules corresponding to agiven color use different exposures. In this way the short exposure timecan be used to reliably capture information corresponding to light(e.g., bright) portions of an image while the long exposure opticalchain module can be used to capture information corresponding to thedarker portions of an image. As discussed above, the sensed pixel valuesfrom the two optical chains can be processed to exclude values generatedby saturated sensors and to combine pixel values corresponding to thesame image area in a manner weighted according to the exposure durationfor pixel value within the acceptable operating range of the opticalchain module's sensors.

While different durations can and often are achieved by controllingsensor exposure times, different filters in different optical chainmodules may, and are, used to achieve different light exposures in someembodiments.

FIG. 8, illustrates an optical chain arrangement used in one panoramiccamera device 8000 in which multiple optical chains and different lensangles are used to capture images that are well suited for combininginto a panoramic image. A1 represents a first non-zero angle, Srepresents a straight or 0 degree angle, and A2 represents a secondnon-zero angle. In one embodiment A1 causes the corresponding camerachain module to capture images to the right of the camera, S causes thecorresponding camera chain module to capture images straight ahead ofthe camera, and A2 causes the corresponding camera chain module tocapture images to the left the camera, from the perspective of the userbehind the camera. In addition to captured images left and right of thecamera it should be appreciated that the optical chain modules capturesome image portion which is also captured by the adjacent optical chainmodule. Thus, the OCMs in columns 803, 805 and 807 capture differentscenes which, while overlapping, can be stitched together to provide anultra wide angle panoramic image. The OCMs in each of rows 811, 813,815, 817, 819, 821, 823 capture different versions of the same scene.

The panoramic camera device 8000 includes multiple optical chain modulescorresponding to each of the left, right and center views. Twenty oneoptical chain modules (seven sets of three) are shown allowing for twooptical chain modules per color (R, G, B) plus a seventh multi-color (R,G, B) optical chain module which can be used to support a preview modeof operation. The multi-color optical chain module may include a sensorwith a multicolor filter, e.g., a Bayer pattern filter, allowing thesingle sensor to capture the multiple colors using different portions ofthe sensor. While the panoramic configuration shown in FIG. 8 isdifferent from that of the non-panoramic camera embodiments previouslydiscussed the exposure control and separate color capture benefitsremain the same as those discussed with regard to the other embodiments.

While FIG. 8 illustrates a particular panoramic embodiment, it should beappreciated that embodiments such as those shown in FIGS. 3 and 4 can,and in sometimes are, used to support taking of panoramic pictures. Inone such embodiment a prism or angled lens is inserted into one or moreoptical chain modules, e.g., by rotation, vertical movement, horizontalmovement and/or a combination of vertical and horizontal movement of aplatter upon which the prism or lens is mounted. The prisms or changesin lens angles change the scene area perceived by one or more opticalchain modules allowing the different optical chain modules to capturedifferent views of a scene which can, and in some embodiments are, usedto generate a panoramic image, e.g., picture. Thus, camera modules usedto capture images corresponding to the same scene which are thencombined to generate a combined image can also be used at a differenttime to capture images corresponding to different views and/or sceneswhich can then be subsequently combined to form a panoramic image, e.g.,photograph.

Accordingly, it should be appreciated that ultra wide angle panoramicimages can be generated using multiple optical chain modules of the typepreviously discussed thereby providing panoramic cameras many of thebenefits of large lens without the need for the camera depth, weight andother disadvantages associated with large lenses.

It should be appreciated that because camera chain modules are separatedfrom one another the multi-optical chain module embodiments of thepresent invention are well suited for stereoscopic image generation andfor generating image depth maps. Accordingly the camera devices of thepresent invention support a wide range of applications and modes ofoperation and provide significant amounts of image data which can beused to support a wide range of post capture image processingoperations.

Having described apparatus and various embodiments, various methodswhich are supported and used in some embodiments will now be discussedwith regard to various flow charts that are included in the presentapplication.

Method 300 of FIG. 9 illustrates one exemplary method of producing atleast one image of a first scene area in accordance with the presentinvention. The processing steps of the method 300 of FIG. 9 will now beexplained in view of the camera device 100 of FIG. 1A.

The method 300 of FIG. 9 starts at start step 302 with the start of thesteps of the method being implemented, e.g., on processor 110. Operationproceeds from start step 302 to step 304. In step 304, user input isreceived to control the capture of at least one image of the first scenearea. The user input is received via input device 106 which may be, andin some embodiments is, a button or touch sensitive screen. In optionalsub-step 306, the user input may, and in some embodiments does, indicatea portion of the first scene area that is to be focused, e.g., in animage to be captured or a combined image to be generated from two ormore captured images. From step 304 processing proceeds to step 308.

In step 308, a plurality of three or more optical chain modules (OCMs),e.g., optical chain modules 130 of FIG. 1A, are operated in parallel tocapture images of the first scene area, said images including at least afirst image of said first scene area, a second image of said first scenearea, and a third image of said first scene area. In some embodimentseach one of the first, second and third optical chain modules captures acorresponding one of the first, second and third image respectively. Insome embodiments, operating a plurality of three or more optical chainmodules in parallel to capture images of the first scene area, saidimages including at least a first image of said first scene area, asecond image of said first scene area, and a third image of said firstscene area includes sub-processing steps 310, 312, and 314.

In sub-step 310 a first optical chain module is operated to capture afirst image 316 of the first scene area. In most, but not all,embodiments, on capture of the first image 316, the image data and otherdata such as camera device configuration information associated with thefirst image is stored in the data/information 120 portion of memory 108for later processing, output or display. In parallel with the processingof sub-step 310 processing of sub-steps 312 and 314 also occur. Insub-step 312 a second optical chain module is operated to capture asecond image 318 of the first scene area. In most, but not all,embodiments on capture of the second image 318, the image data and otherdata such as camera device configuration information associated with thesecond image is stored in the data/information 120 portion of memory 108for later processing, output or display. In sub-step 314 a third opticalchain module is operated to capture a third image 320 of the first scenearea. In most, but not all, embodiments on capture of the third image320, the image data and other data such as camera device configurationinformation associated with the third image is stored in thedata/information 120 portion of memory 108 for later processing, outputor display. Processing then proceeds from step 308 to step 322.

In some embodiments, each optical chain module of the plurality ofoptical chain modules includes a lens and the lenses of the plurality ofthe optical chain modules are arranged along a circle. For example, whenthere are three optical chain modules, i.e., a first optical chainmodule, a second optical chain module, and a third optical chain module,the first optical chain module includes a first lens, the second opticalchain module includes a second lens, and the third optical chain moduleincludes a third lens. The first, second and third lenses are arrangeduniformly along a circle, e.g. on the vertices of an equilateraltriangle. In some embodiments the camera device 100 includes a fourthoptical chain module including a fourth lens, said fourth lens beingpositioned in the center of the circle. Each of the first, second, thirdand fourth lens may be, and in some embodiments of the present inventionare, the outer lens of each of their respective optical chain modulesand are all positioned in the same plane. More generally, in someembodiments of the present invention, there are a plurality of N opticalchain modules each including a lens. N−1 lenses of the plurality ofoptical chain modules are arranged along a circle with Nth lens beingpositioned in the center of the circle. FIG. 1B illustrates and exampleof a camera device 100 with seven optical chain modules which include 7outer lenses shown as circles, i.e., OCM1, OCM2, OCM3, OCM4, OCM5, OCM6,and OCM7. The outer lens of optical chain modules OCM 1, OCM2, OCM3,OCM4, OCM5, and OCM6 are arranged along a circle and the outer lens ofoptical chain module OCM7 is positioned in the center of the circle.

In some embodiments of the present invention, the first optical chainmodule includes in addition to the first lens an image sensor referredto as a first image sensor. In some embodiments of the presentinvention, the second optical chain module includes an image sensorreferred to as a second image sensor. In some embodiments of the presentinvention, the third optical chain includes an image sensor referred toas a third image sensor. In some embodiments of the present inventionthe plurality of lenses of the plurality of optical chain modules aremounted in a cell phone housing with the plurality of lenses oriented inthe same direction and in the same plane of the housing. For example inthe case of three optical chain modules, in some embodiments of thepresent invention, the first, second and third lenses of the first,second, and third optical chain modules respectively are mounted in acell phone housing and are oriented in the same direction and in thesame plane of the housing.

In step 322, said first, second, and third images are processed byprocessor 110 to generate a first combined image 326 of said first scenearea. In some embodiments, including those embodiments of the presentinvention in which user input is received indicating a portion of thefirst scene area to be focused in the combined image, step 322 may, andin some embodiments does, include sub-step 324 wherein pixel positionson at least one of said first, second, and third images is shifted priorto generating said first combined image to align the portion of thefirst scene to be focused. Processing then proceeds to step 328 wherethe generated combined image is stored in data/information 120 of memory108, e.g., for potential later display, output from the camera device,and/or additional processing and/or displayed on display 102 of cameradevice 100.

In some embodiments, processing step 322 and/or sub-step 324 areperformed on an external device such as a computer. In such cases, thefirst, second and third images are outputted from the camera device 100via transceiver 114 to the external computer for processing to generatethe first combined image 326. The first combined image may then bestored in memory associated with the external device and/or displayed ona display associated with the external computer. In some embodiments ofthe present invention, the first combined image of the first scene areaincludes the same or fewer pixel values than either of said first,second or third images.

From step 328 processing proceeds to step 304 where processing continuesand the method is repeated.

In some embodiments of the present invention, the size of the diameterof the first, second and third lens of the first, second, and thirdoptical chain modules respectively are the same and the sensors of thefirst, second and third optical chain modules have the same number ofpixels. In other embodiments of the present invention, one or moreoptical chain modules may, and in some embodiments do, have lenses withdifferent diameter sizes and/or sensors with different numbers ofpixels. In some embodiments of the present invention, the first, secondand third lenses of the first, second and third optical chain modulesrespectively, are less than 2 cm in diameter and each of the first,second and third image sensors of the first, second and third opticalchain modules support at least 8 Mpixels. In some embodiments of thepresent invention, the first and second lenses are each less than 2 cmin diameter and each of the first and second image sensors support atleast 5 Mpixels. However in many embodiments the image sensors support 8Mpixels or even more and in some embodiments the lenses are larger than2 cm. Various combinations of lens and sensors may be used with avariety of lens sizes being used for different optical chains in someembodiments. In addition different optical chains may use lenses withdifferent shapes, e.g., while the lens may be a spherical lens theperimeter of the lens may be cut into one of a variety of shapes. In oneembodiment, lenses of different optical chain modules are shaped andarranged to minimize gaps between lenses. Such an approach can have theadvantage of resulting in a smoother blur with regard to portions ofcaptured images which are out of focus when combining images captured bydifferent optical chain modules and result in an overall image whichmore closely approximates what might be expected had a single large lensbeen used to capture the scene shown in the combined image.

In accordance with some aspects of the present invention, the diametersize and arrangement of the lenses of the plurality of optical modulesmay and do vary. Similarly the number of pixels supported by the sensorsof each of the plurality of optical modules may also vary for exampledepending on the desired resolution of the optical chain module.

In some embodiments, different shifts are used for different portions ofthe scene to create a single composite image. In some embodiments, thegenerated combined image is a panoramic image.

In various embodiments, the optical chain modules are independentlyfocused to the same focal distance. In some embodiments, the opticalchain modules are focused together. In some such embodiments, theoptical chain modules are focused together by moving a platter on whichlenses corresponding to different optical chains are mounted.

Method 400 of FIG. 10 illustrates an embodiment of a method of producingat least one image of a first scene area in accordance with the presentinvention. The method 400 achieves enhanced sensor dynamic range bycombining images captured through the operation of two or more opticalchain modules using different exposure times. The processing steps ofthe method 400 of FIG. 10 will now be explained in view of the cameradevice 100 of FIG. 1A. For ease of explanation of the method 400, itwill be assumed that the plurality of optical chain module 130 of cameradevice 100 of FIG. 1A includes two optical chain modules and in someembodiments an optional third optical chain module which will bereferred to as a first, second and third optical chain modulerespectively.

The method 400 of FIG. 10 starts at start step 402 with the start of thesteps of the method being implemented, e.g., on processor 110. Operationproceeds from start step 402 to step 404. In step 404, one of aplurality of optical chain modules of the camera device is operated tocapture an image which will be referred to herein as a fourth image ofthe first scene. For example, one of said first, second or optionalthird optical chain modules may be, and in some embodiments is, operatedto capture the fourth image. This fourth image is captured prior tocapturing the first, second or third images which will be discussed inconnection with step 410 below.

Processing then proceeds to step 406 where the fourth image is displayedon the display 102 of the camera device 100. By displaying the fourthimage on the display of the camera device 100 a user can aim the cameradevice and target the first scene area for which the user wants tocapture an image. In some embodiments, the fourth image is also storedin data/information 120 of memory 108. Processing then proceeds fromstep 406 to step 408.

In step 408, user input is received to control the capture of an imageof the first scene area. The user input is received via input device 106which may be, and in some embodiments is, a button or touch sensitivescreen. For example, the user may touch a portion of the touch sensitivescreen on which the fourth image is shown to focus the camera on aportion of the scene for which an image is to be captured. From step 408processing proceeds to step 410 where the plurality of optical chainmodules 130 are operated in parallel to capture images of the firstscene area.

Step 410 includes sub-steps 412, 414, and optional sub-step 416. Insub-step 412, a first optical chain module is operated to capture afirst image 418 of the first scene area using a first exposure time. Insub-step 414, a second optical chain module is operated to capture asecond image 420 of the first scene area using a second exposure time,at least said first and said second exposure times being of differentduration but overlapping in time. In some embodiments, an optionalsub-step 416 is performed wherein a third optical chain module isoperated to capture a third image 422 of the first scene area using athird exposure time. In some embodiments, the third exposure time isdifferent than the first and second exposure times. Additional opticalchain modules may be, and in some embodiments are, used to captureadditional images of the first scene area with the additional opticalchain modules using the same or different exposure times as the first,second or third exposure times so as to obtain additional image data forthe first scene area. Sub-steps 412, 414, and optional sub-step 416 areperformed in parallel so that multiple images of the first scene arecaptured in parallel with different exposure times. The first, secondand optional third captured images may be, and in some embodiments are,stored in data/information 120 of memory section 108 to be available forlater use such as for example in later steps of the method forgenerating a combined image of the first scene area, or for display oroutputting of images.

In some embodiments, in step 404 the operation of one of the first,second and third optical chain modules to capture the fourth image ofthe first scene area uses a fourth exposure time different from saidfirst, second and third exposure times. Once again step 404 occurs priorto the step 410 as the fourth image is displayed on the display 102 sothe user can utilize the displayed image to target the scene area to becaptured by the first, second and optional third images.

Operation of the method proceeds from step 410 to step 424. In step 424the captured images, that is the first and second images, are processedto generate a first combined image of the first scene area 430. In thoseembodiments in which the optional third image was captured optionalsub-step 428 is performed wherein the third image in addition to thefirst and second image is also processed to generate the first combinedimage of the scene area 430.

In some embodiments step 424 is accomplished using sub-step 426 whereinsaid processing of said first and second images and optionally saidthird image to generate a first combined image of the first scene areaincludes combining weighted pixel values of said first image, secondimage, and optional third image.

The weighting of the pixel values may, and in some embodiments is afunction of exposure times. Thus, at least in some embodiments, a pixelvalue of the combined image is generated by weighting and summing apixel value from each of the first, second and third images, where thepixel value from the first image is weighted according to the firstexposure time used to capture the first image, the pixel value from thesecond image is weighted according the second exposure time used tocapture the second image and the pixel value from the third image isweighted according to the third exposure time used to capture the thirdimage.

Operation proceeds from step 424 to step 432. In step 432, the generatedfirst combined image of the first scene area is stored indata/information 120 of memory 108 and/or displayed on the display 102,e.g., touch sensitive display of the camera device 100.

Operation proceeds from step 432 to step 404 where processing continuesand the method is repeated.

In some embodiments of the present invention step 424 is performed on anexternal device such as a computer that is coupled to the camera device100 via the transceiver interface 114. In such embodiments the first,second and optional third images are transmitted to the external devicevia the transceiver interface 114 where the step 424 is performed. Step432 is then typically performed by the external device with the combinedimage 430 being stored in memory associated with the external deviceand/or displayed on a display associated with the external device.

Method 400 may be, and in some embodiments is, implemented on a varietyof devices including for example, a camera or a mobile device such as amobile cellular telephone or a tablet.

In some embodiments, at least some of the optical chain modules includesingle color filters. For example, in one embodiment, the first opticalchain module includes a red filter, the second optical chain moduleincludes a green filter, the third optical chain module includes a bluefilter. In some such embodiments, at least two optical chain modules areprovided for each color for which a single color filter is used. Forexample in one embodiment, the plurality of optical chains modulesinclude two optical chain modules with a red filter, two optical chainmodules with a green filter and two optical chain modules with a bluefilter. In some embodiments, different optical chain modules havingsingle color filters corresponding to the same color have differentexposure times. In some embodiments, the combined image is generatedusing captured images of the first scene area from: (i) an optical chainmodule including a first color filter and a using first exposure time,(ii) an optical chain including a second color filter and using a firstexposure time, (iii) an optical chain including a third color filter andusing a first exposure time, (iv) an optical chain module including afirst color filter and a using second exposure time, (ii) an opticalchain including a second color filter and using a second exposure time,(iii) an optical chain including a third color filter and using a secondexposure time. In some such embodiments, the first color is red; thesecond color is green; and the third color is blue; the first exposuretime is a short exposure time and the second exposure time is a longexposure time.

In some embodiments, at least some optical chain modules do not includeany color filters.

Method 500 of FIG. 11 illustrates an embodiment of a method of producingat least one image of a first scene area in accordance with the presentinvention. The method 500 achieves enhanced sensor dynamic range bycombining images captured through the operation of two or more opticalchain modules using different exposure times. The processing steps ofthe method 500 of FIG. 11 will now be explained in view of the cameradevice 100 of FIG. 1A. For ease of explanation of the method 500, itwill be assumed that the plurality of optical chain module 130 of cameradevice 100 of FIG. 1A includes two optical chain modules and in someembodiments an optional third optical chain module which will bereferred to as a first, second and third optical chain modulerespectively. Method 500 is similar to method 400 but implements thecapture of the fourth image and display of the fourth image after thefirst, second and third images have been captured. In this way the userof the device is able to see on the display the first scene area thatwas captured in the first, second and optional third image and whichwill be processed to generate a combined image.

The method 500 of FIG. 11 starts at start step 502 with the start of thesteps of the method being implemented, e.g., on processor 110. Operationproceeds from start step 502 to step 504. In step 504, user input isreceived to control the capture of the image of the first scene area.The user input is received via input device 106 which may be, and insome embodiments is, a button or touch sensitive screen. From step 504processing proceeds to step 506 where the plurality of optical chainmodules 130 are operated in parallel to capture images of the firstscene area.

Step 506 includes sub-steps 510, 512, and optional sub-step 514. Insub-step 510, a first optical chain module is operated to capture afirst image 516 of the first scene area using a first exposure time. Insub-step 512, a second optical chain module is operated to capture asecond image 518 of the first scene area using a second exposure time,at least said first and said second exposure times being of differentduration but overlapping in time. In some embodiments, an optionalsub-step 514 is performed wherein a third optical chain module isoperated to capture a third image 520 of the first scene area using athird exposure time. In some embodiments, the third exposure time isdifferent than the first and second exposure times. Additional opticalchain modules may be, and in some embodiments are, used to captureadditional images of the first scene area with the additional opticalchain modules using the same or different exposure times as the first,second or third exposure times so as to obtain additional image data forthe first scene area and thereby enhancing the effective sensor dynamicrange of the camera device. Sub-steps 510, 512, and optional sub-step514 are performed in parallel so that multiple images of the first sceneare captured in parallel with different exposure times. The first,second and optional third captured images may be, and in someembodiments are, stored in data/information 120 of memory section 108 tobe available for later use such as for example in later steps of themethod for generating a combined image of the first scene area, or fordisplay or outputting of the images. Operation proceeds from step 506 tosteps 522 and 528.

In step 522, one of said first, second and optional third optical chainmodules is operated to capture a fourth image 524 of the first scenearea after capturing one of said first, second and third images. Whilein this particular embodiment the fourth image is captured after thefirst, second and third images, in some embodiments one of the first,second and third images is used as the fourth image. In some embodimentsa fourth exposure time different from said first, second and thirdexposure times is used to capture the fourth image 524. The fourth imagemay be, and in some embodiments is stored in data/information 120 ofmemory 108 for potential later use, output or display. Processingproceeds from step 522 to step 526. In step 526, the fourth image of thefirst scene area is displayed on display 102 of the camera device, e.g.,a touch sensitive screen so that a user of the camera device can see animage of the first scene area that was captured by the first, second andoptional third images. Processing proceeds from step 526 to step 504where processing associated with the method continues as the method isrepeated.

Returning to step 528, in step 528 the first and second images areprocessed to generate a first combined image of the first scene area534. In those embodiments in which the optional third image was capturedoptional sub-step 532 is performed wherein the third image in additionto the first and second images is also processed to generate the firstcombined image of the scene area 534.

In some embodiments step 528 is accomplished using sub-step 530 whereinsaid processing of said first and second images and optionally saidthird image to generate a first combined image of the first scene areaincludes combining weighted pixel values of said first image, secondimage, and optional third image. The weighting of the pixel values may,and in some embodiments is a function of exposure times. Thus, at leastin some embodiments, a pixel value of the combined image is generated byweighting and summing a pixel value from each of the first, second andthird images, where the pixel value from the first image is weightedaccording to the first exposure time used to capture the first image,the pixel value from the second image is weighted according the secondexposure time used to capture the second image and the pixel value fromthe third image is weighted according to the third exposure time used tocapture the third image.

Operation proceeds from step 528 to step 536. In step 536, the generatedfirst combined image of the first scene area is stored indata/information 120 of memory 108 and/or displayed on the display 102,e.g., the touch sensitive display of the camera device 100.

Operation proceeds from step 536 to step 504 where processing continuesand the method is repeated.

In some embodiments of the present invention step 528 is performed on anexternal device such as a computer that is coupled to the camera device100 via the transceiver interface 114. In such embodiments the first,second and optional third images are transmitted to the external devicevia the transceiver interface 114 where the step 528 is performed. Step536 is then typically performed by the external device with the combinedimage 534 being stored in memory associated with the external deviceand/or displayed on a display associated with the external device.

Method 500 may be, and in some embodiments, is implemented on a varietyof devices including for example, a camera or a mobile device such as amobile cellular telephone or a tablet.

The use of an external computer to perform some or a part of theprocessing of the first, second and optional third images allows for theuse of computational more complex algorithms as the external computermay be, and in some embodiments does have, a computationally morepowerful processing capability than the camera device 100.

In some embodiments, at least some of the optical chain modules includesingle color filters. For example, in one embodiment, the first opticalchain module includes a red filter, the second optical chain moduleincludes a green filter, the third optical chain module includes a bluefilter. In some such embodiments, at least two optical chain modules areprovided for each color for which a single color filter is used. Forexample in one embodiment, the plurality of optical chains modulesinclude two optical chain modules with a red filter, two optical chainmodules with a green filter and two optical chain modules with a bluefilter. In some embodiments, different optical chain modules havingsingle color filters corresponding to the same color have differentexposure times. In some embodiments, the combined image is generatedusing captured images of the first scene area from: (i) an optical chainmodule including a first color filter and a using first exposure time,(ii) an optical chain including a second color filter and using a firstexposure time, (iii) an optical chain including a third color filter andusing a first exposure time, (iv) an optical chain module including afirst color filter and a using second exposure time, (ii) an opticalchain including a second color filter and using a second exposure time,(iii) an optical chain including a third color filter and using a secondexposure time. In some such embodiments, the first color is red; thesecond color is green; and the third color is blue; the first exposuretime is a short exposure time and the second exposure time is a longexposure time.

In some embodiments, at least some optical chain modules do not includeany color filters. For example, in one particular embodiment, opticalchain modules OCM 171 and OCM 173 do not include color filters. Howeverin other embodiments, OCM 171 and OCM 173 each include a color filter.

Method 600 of FIG. 12 illustrates an embodiment of a method of producingat least one color image of a first scene area in accordance with thepresent invention. The method 600 uses color filters in connection withcombining two or more images of a first scene area to obtain a colorimage of the first scene area. The processing steps of the method 600 ofFIG. 12 will now be explained in view of the camera device 100 of FIG.1A. For ease of explanation of the method 600, it will be assumed thatthe plurality of optical chain module 130 of camera device 100 of FIG.1A includes two optical chain modules and in some embodiments anoptional third and/or fourth optical chain module which will be referredto as a first, second, third and fourth optical chain modulerespectively.

The method 600 of FIG. 12 starts at start step 602 with the start of thesteps of the method being implemented, e.g., on processor 110. Operationproceeds from start step 602 to step 604. In optional step 604, a fourthoptical chain module of the camera device is operated to capture animage, e.g., a image referred to herein as a fourth image of a firstscene area using a multi-color filter. This fourth image is capturedprior to capturing the first, second or third images which will bediscussed in connection with step 610 below.

Processing then proceeds to optional step 606 where the fourth image isdisplayed on the display 102 of the camera device 100. By displaying thefourth image on the display of the camera device 100 a user can aim thecamera device and target the first scene area for which the user wantsto capture an image. In some embodiments, the fourth image is alsostored in data/information 120 of memory 108. Processing then proceedsfrom step 606 to step 608.

In step 608, user input is received to control the capture of an imageof the first scene area. The user input is received via input device 106which may be, and in some embodiments is, a button or touch sensitivescreen. For example, the user may touch a portion of the touch sensitivescreen on which the fourth image is shown to focus the camera on aportion of the scene for which an image is to be captured. From step 608processing proceeds to step 610 where the plurality of optical chainmodules 130 are operated in parallel to capture images of the firstscene area.

Step 610 includes sub-steps 612, 614, and optional sub-step 616. Insub-step 612, a first optical chain module is operated to capture afirst image 618 of the first scene area using a first color filter. Insub-step 614, a second optical chain module is operated to capture asecond image 620 of the first scene area using a second color filter,said first and said second color filters corresponding to a first colorand a second color respectively. Said first and said second colors beingdifferent colors. In some embodiments, said first and second colorfilters are single color filters which correspond to said first andsecond colors, respectively. In some embodiments, an optional sub-step616 is performed wherein a third optical chain module is operated tocapture a third image 622 of the first scene area using a third colorfilter. In some embodiments, the third color filter corresponds to acolor that is different from said first and second colors. In someembodiments the third color filter is a single color filter whichcorresponds to said third color. Additional optical chain modules maybe, and in some embodiments are, used to capture additional images ofthe first scene area with the additional optical chain modules using thesame or different color filters as the first, second or third colorfilters so as to obtain additional image data for the first scene area.Sub-steps 612, 614, and optional sub-step 616 are performed in parallelso that multiple images of the first scene area are captured in parallelwith different color filters. The first, second and optional thirdcaptured images may be, and in some embodiments are, stored indata/information 120 of memory section 108 to be available for later usesuch as for example in later steps of the method for generating acombined image of the first scene area, or for display or outputting ofimages. In some embodiments of the present invention, the first opticalchain module includes a first lens and a first image sensor and thesecond optical module includes a second lens and a second image sensorand the optional third optical chain module includes a third lens and athird image sensor. In some embodiments, said first and said secondimage sensors are of the same resolution. In some embodiments of thepresent invention, said optional third image sensor of said thirdoptical chain module has the same resolution as the first and secondimage sensors. In some embodiments of the present invention, the fourthoptical chain module includes a fourth lens and a fourth image sensor.In some embodiments of the present invention the fourth image sensor isof the same resolution as the first and second image sensor. In someembodiments of the present invention, the first, second and third lensesof the first, second and third optical chain modules are arranged in acircle, and the fourth lens of the fourth optical chain is arranged inthe center of the circle.

Operation of the method proceeds from step 610 to step 624. In step 624the captured images, that is the first and second images, are processedto generate a first combined image of the first scene area 630. In thoseembodiments in which the optional third image was captured optionalsub-step 628 is performed wherein the third image in addition to thefirst and second images is also processed to generate the first combinedimage of the scene area 630. In some embodiments the fourth image of thefirst scene area is also processed with the first, second and thirdimages to generate the first combined image of the first scene area.

Operation proceeds from step 624 to step 632. In step 632, the generatedfirst combined image of the first scene area is stored indata/information 120 of memory 108 and/or displayed on the display 102,e.g., a touch sensitive display of the camera device 100.

Operation proceeds from step 632 to step 604 where processing continuesand the method is repeated.

In some embodiments of the present invention step 624 is performed on anexternal device such as a computer that is coupled to the camera device100 via the transceiver interface 114. In such embodiments the first,second and optional third images are transmitted to the external devicevia the transceiver interface 114 where the step 624 is performed. Step632 is then typically performed by the external device with the combinedimage 630 being stored in memory associated with the external deviceand/or displayed on a display associated with the external device.

Method 600 may be, and in some embodiments, is implemented on a varietyof devices including for example, a camera or a mobile device such as amobile cellular telephone or a tablet.

In some embodiments of the present invention, each image is presented asit is captured on the display or in the case of a combined image whensaid image has been generated.

In some embodiments of the present invention, each of the capturedimages, e.g., the first, second, third, and fourth images may be, andis, displayed on the display 102 of the camera device 100 as it iscaptured along with one or more combined images that are formed byprocessing and/or combining the first, second, third and/or fourthimages. In some embodiments of the present invention, each of the imagesmay be, is shown, in a separate portion of the display with the size ofthe image being adjusted so that each image displayed is shown in itsentirety. In some embodiments of the present invention, a caption isautomatically placed under each image as it displayed on the screen. Insome embodiments of the present invention, the caption includes thenumber of the image or an indication that it is a combined image, e.g.,image 1, image 2, image 3, image 4, combined image from image 1, 2, 3,and 4. In some embodiments of the present invention, each image ispresented as it is captured on the display or in the case of a combinedimage when said image has been generated. The images may be arranged ina variety of ways on the display 102 after capture and theaforementioned embodiments are only meant to be exemplary in nature.

In some embodiments of the present invention, the image generated bycombining the images captured from two or more of the optical chainmodules is displayed for targeting purposes so that the user may provideinput to control the capture of the image of the scene area and/or theobject in the scene upon which the combined image should be focused.

The FIG. 13 assembly of modules 1300 may, and in some embodiments is,used to process data for example first, second, third and fourth imagesand associated data, and storing and displaying images. Assembly ofmodules 1300 may be included in an exemplary apparatus, e.g., a cameradevice, e.g., camera device 100 of FIG. 1A, camera device 200 of FIG. 2,camera device 60 of FIG. 4, camera device 1500 of FIG. 15, camera device1605 of FIG. 16, camera device 1705 of FIG. 17, camera device 1801 ofFIG. 18, and/or camera device 1905 of FIG. 19, in accordance with anexemplary embodiment.

In some embodiments, assembly of modules 1300 is included in memory inan exemplary camera device, e.g., memory 108 of camera device 100 ofFIG. 1A, memory 213 of camera device 200 of FIG. 2, memory 73 of cameradevice 60 of FIG. 4, memory of camera device 1500 of FIG. 15, memory ofcamera device 1605 of FIG. 16, memory in camera device 1705 of FIG. 17,memory in camera device 1801 of FIG. 18, and/or memory of camera device1901 of FIG. 19. For example assembly of modules 1300 may be included aspart of assembly of modules 118 of memory 108 of camera device 100 ofFIG. 1.

In some embodiments, assembly of modules 1300 is implemented inhardware. In some embodiments, assembly of modules 1300 is implementedas software. In some embodiments, assembly of modules 1300 isimplemented as a combination of hardware and software.

In some embodiments, all or part of assembly of modules 1300 may beincluded as part of a processor, e.g., as part of processor 110 ofcamera device 100 of FIG. 1A.

In the FIG. 13 example, the assembly of modules 1300 includes a imageprocessing module 1302, a display module 1304, and a storage module1306. The modules implemented one or more of the previously discussedimage processing steps and may include a variety of sub-modules, e.g.,an individual circuit, for performing an individual step of method ormethods being implemented. Image processing module 1302 is configuredto: (1) process said first, second and third images to generate a firstcombined image of a first scene area, (2) receive user input indicatinga portion of the first scene area to be focused in the first combinedimage; and/or (3) shift pixel positions on at least one of said firstsecond and third images prior to generating said first combined image toalign the portion of the first scene area to be focused, as part ofprocessing said first, second, and third images to generate a firstcombined image; and (4) weight and sum a combination of pixel values ofthe first and second images corresponding to the same portion of thefirst scene area as a function of the first and second exposure timesrespectively and summing the weighted pixel values. In some embodimentsimage processing module 1302 is further configured to process said thirdimage to generate said first combined image of said first scene areafrom the third image in addition to said first and second images.

Display module 1304 is configured to display said fourth image on saiddisplay and configured to display said combined image on said display.Storage module 306 is configured to store or more or said first image,said second image, said third image, said fourth image and said combinedimage in memory.

FIG. 14 illustrates a computer system which can be used for postprocessing of images captured using a camera device. The computer system1400 includes a display 1402, Input/Output (I/O) interface 1412,receiver 1404, input device 1406, transceiver interface 1414, processor1410 and memory 1408. Memory 1408 includes a first portion 1424including data/information 1420 and an assembly of modules 1418, and asecond portion 1426 including storage 1422. The memory 1408 is coupledto the processor 1410, I/O interface 1412 and transceiver interface 1414via bus 1416 through which the elements of the computer system 1400 canexchange data and can communicate with other devices via the I/Ointerface 1412 and/or interface 1414 which can couple the system 1400 toa network and/or camera apparatus. It should be appreciated that viainterface 1414 image data can be loaded on to the computer system 1400and subject to processing, e.g., post capture processing. The images maybe stored in the storage portion 1422 of memory 1408 for processing.Data/information 1420 includes, e.g., intermediate processing data andinformation and criteria used for processing e.g., weightinginformation, exposure time information, etc. The assembly of modules1418 includes one or more modules or routines which, when executed bythe processor 1410, control the computer system to implement one or moreof the image processing operations described in the present application.The output of multiple optical receiver chains can be, and in someembodiments is, combined to generate one or more images. The resultingimages are stored in the storage portion of the memory 1408 prior tobeing output via the network interface 1414, though another interface,or displayed on the display 1402. Thus, via the display 1402 a user canview image data corresponding to one or more individual optical chainmodules as well as the result, e.g., image, generated by combining theimages captured by one or optical chain modules.

FIG. 15 illustrates a frontal view of an apparatus 1500 implemented inaccordance with one embodiment of the present invention whichincorporates multiple optical chain modules. Camera device 1500 includesfour optical chains OCM 1 1502, OCM 2 1504, OCM 3 1506 and OCM 4 1508.The outer lens of OCM 1, OCM 2, OCM 3 and OCM 4, OCM 1 L1 1510, OCM 2 L11512, OCM 3 L1 1514, OCM 4 L1 1516, respectively, being shown as solidline circles with a frontal view. OCM 1 1502 including a red filterelement 1518, OCM 2 1504 including a green filter element 1520, OCM 31506 including a blue filter element 1522. Optical chain module 4 1508passes all three colors and includes a sensor with a multi-color filterelement 1524, e.g., a Bayer filter. The optical chain modules (1502,1504, 1506, 1508) may be the same as or similar to those previouslydescribed in FIGS. 1-3.

FIG. 16 illustrates a frontal view of the outer lenses of an apparatus1605, e.g., a camera device, implemented in accordance with oneembodiment of the present invention which incorporates multiple opticalchain modules and which is designed to have little or no gaps betweenthe outer most lenses of the different optical chain modules. The outermost lenses may be the aperture stop lenses in the FIG. 16 embodiment.Apparatus 1605 of FIG. 16 includes 7 optical chain modules OCM1, OCM2,OCM3, OCM4, OCM5, OCM6 and OCM7 with the outer lens plane correspondingto lenses L1 as viewed from the front of the camera device being shownin FIG. 16.

The 7 optical chain modules are, e.g., optical chain modules (OCM 1 161,OCM 2 161′, OCM 3 161″, . . . , OCM 7 161′″, of FIG. 1D with the outerlens (OCM 1 L1 162, OCM 2 L1 162′, OCM 3 L1 162″, . . . , OCM 7 L1162′″) being outer lenses (OCM 1 L1 1607, OCM 2 L1 1609, OCM 3 L1 1611,. . . , OCM 7 L1 1619) of FIG. 16, respectively.

The outer lenses L1 of optical chain modules 1, 2, 3, 4, 5, and 6, OCM 1L1 1607, OCM 2 L1 1609, OCM 3 L1 1611, OCM 4 L1 1613, OCM 5 L1 1615, OCM6 L1 1617, are positioned so as to surround the outer lens L1 of theoptical chain module 7, OCM 7 L1 1619. The outer lens L1 of the opticalchain module 7 1619 being formed in the shape of a hexagon, i.e., a sixsided polygon. The outer lenses L1 of optical chain modules 1, 2, 3, 4,5 and 6 (1607, 1609, 1611, 1613, 1615, 1617) being of same shape andsize and when combined with lens L1 of optical module 7 (1619) forming acircle. The optical center of each lens L1 of optical chain modules (OCM1 L1 1607, OCM 2 L1 1609, OCM 3 L1 1611, OCM 4 L1 1613, OCM 5 L1 1615,OCM 6 L1 1617) shown as a dark solid dot (1612, 1623, 1625, 1627, 1629,1631) on the dashed circle 1651. The optical center of lens L1 1619 ofoptical chain module 7 shown as a dot 1633 in the center of the hexagonand also in center of the dashed line 1651. A block separator or otherlight block may be used between the lenses to stop light leakage betweenthe different lenses. The dots (1621, 1623, 1625, 1627, 1629, 1631,1633) in FIG. 16 represent the optical center of the individual lenses(1607, 1609, 1611, 1613, 1615, 1617, 1619), respectively. In someembodiments each outermost lens is a round convex lens with itsparameter cut to the shape shown in FIG. 16 so that the lenses fightclosely together. The little or no gap between the front lenses, e.g.,the total area of the gap between the lenses occupies less than 5% ofthe total area of the front area of the lens assembly, e.g., circleshown in FIG. 16, occupied by the lenses when assembled together. Thelack of or small size of the gaps facilitates generating combined imageswith a desirable bokehs or blurs in the combined image with regard toimage portions which are out of focus, e.g., in some cases without theneed for extensive and potentially complex processing to generate thecombined image.

In FIG. 16, circle 1603 represents a circular aperture for the cameradevice 1605. In other embodiments, the aperture for the camera device1605 is a polygon shaped aperture. The plurality of lenses (1607, 1609,1611, 1613, 1615, 1615, 1617, 1619) are configured to partition theaperture 1603 into a plurality of light capture areas (1641, 1643, 1645,1647, 1649, 1651, 1653), occupying substantially the entire area of thefirst aperture.

In some embodiments, the seven optical chains included in camera device1605 are the N optical chains (161, 161′, 161″ . . . , 161′″), whereN=7, where the outer lenses configuration of FIG. 16 is used. Forexample, OCM 1 L1 162 of FIG. 1D is OCM L1 1607 of FIG. 16, OCM 2 L1162′ of FIG. 1D is OCM 2 L1 1609 of FIG. 16, OCM 3 L1 162″ of FIG. 1D isOCM 3 L1 1611 of FIG. 16, . . . , and OCM N L1 162′″ of FIG. 1D is OCM 7L1 1619 of FIG. 16.

In various embodiments, the sensor included in each optical chain incamera device 1605 is a semiconductor sensor. In various embodiments,first aperture of camera device 1605 is one of a circular or polygonshaped aperture. The first aperture of camera device 1605 corresponds tocircle 1603. In some other embodiments, the first aperture correspondsto a polygon, e.g., a polygon approximately the same size as circle1603. In some embodiments, the polygon fits inside circle 1603. In someembodiments, the polygon is a regular polygon.

The lenses (1607, 1609, 1611, 1613, 1615, 1617) in said plurality oflenses (1607, 1609, 1611, 1613, 1615, 1617, 1619) which are arrangedalong the perimeter of said first aperture 1603 have optical centers(1621, 1623, 1625, 1627, 1629, 1631) which are arranged along a circle1651. The lenses (1607, 1609, 1611, 1613, 1615, 1617) in said pluralityof lenses (1607, 1609, 1611, 1613, 1615, 1617, 1619) which are arrangedalong the perimeter of said first aperture 1603 have optical centers(1621, 1623, 1625, 1627, 1629, 1631) which form the vertices (corners)of a regular polygon 1655.

The plurality of lenses (1607, 1609, 1611, 1613, 1615, 1617, 1619)includes at least one inner lens 1619 in addition to said lenses (1607,1609, 1611, 1613, 1615, 1617) arranged along the perimeter of said firstaperture 1603. The plurality of lenses (1607, 1609, 1611, 1613, 1615,1617, 1619) includes a total of six lenses (1607, 1609, 1611, 1613,1615, 1617) along the perimeter of said first aperture 1603 and a singlelens (1619) in the center of said six lenses (1607, 1609, 1611, 1613,1615, 1617) arranged along the perimeter of said first aperture 1603.

The non-circular aperture of each of said plurality of lenses (1607,1609, 1611, 1613, 1615, 1617, 1619) is an aperture stop in acorresponding optical chain.

Each lens in said plurality of lenses (1607, 1609, 1611, 1613, 1615,1617, 1619) is part of a corresponding optical chain, each individualoptical chain includes a separate sensor for capturing an imagecorresponding to said individual optical chain.

Apparatus 1605, e.g., a camera device, further includes a module, e.g.,module 1302 of FIG. 13, for combining images captured by separateoptical chains into a single combined image. In various embodiments, thecombining images, e.g., performed by module 1302, includes a shift andadd based on the position of lenses in said plurality of lenses (1607,1609, 1611, 1613, 1615, 1617, 1619).

Camera device 1605 further includes additional elements shown in FIG. 1Aincluding a processor, a memory and a display.

FIG. 17 illustrates a frontal view of the outer lenses of an apparatus1705 implemented in accordance with one embodiment of the presentinvention which incorporates multiple optical chain modules and outerlenses, e.g., the aperture stop lens for each of the correspondingoptical chains, arranged to have non-uniform spacing between the opticalcenters of the lenses. Thus the FIG. 17 embodiment is similar to theFIG. 16 embodiment but with non-uniform spacing of the optical centersof lenses along the outer parameter of the lens assembly. Thenon-uniform spacing facilitates depth of field determinationsparticularly when performing block processing and the entire field ofview may not be under consideration when processing a block orsub-portion of the captured field of view. The optical chain modulesshown in FIGS. 16 and 17 are the same or similar to those previouslydescribed with reference to FIG. 3 but differ in terms of lens shape,size and/or configuration. The dots (1721, 1723, 1725, 1727, 1729, 1731,1733) in FIG. 17 represent the optical center of the individual lenses(1707, 1709, 1711, 1713, 1715, 1717, 1719), respectively.

FIG. 18 illustrates another exemplary camera device 1801 including aplurality of first through fifth optical chain modules (1890, 1891,1892, 1893, 1894) each of which includes an outer lens (1813, 1815,1817, 1819, 1821), respectively, represented as a circle on the outerlens platter 1803. Each outer lens (1813, 1815, 1817, 1819, 1821) has anoptical axis (1805, 1806, 1807, 1808, 1809), respectively. The opticalaxis (1805, 1806, 1807, 1808, 1809) is represented by an X, indicatingthat the axis goes down into the lens (1813, 1815, 1817, 1819, 1821).The optical axis (1805, 1806, 1807, 1808, 1809), are parallel to eachother. In both FIGS. 18 and 19 arrows made of dashed lines represent thepath of light for the corresponding optical chain module after lightwhich entered the outer lens along the optical axis of the outer lens isredirected by the mirror or other light redirection device. Thus, thearrows represents the direction and general light path towards thesensor of the optical chain to which the arrow corresponds. In variousembodiments, the image deflection element, e.g., a mirror, of theoptical chain changes the direction of the optical rays passing alongthe optical axis of the outer lens by substantially 90 degrees to directthe optical rays passing along the optical axis onto the sensor. Forexample, with regard to optical chain 1890, the image deflection element1823, e.g., a mirror, of the optical chain 1890 changes the direction ofthe optical rays passing along the optical axis 1805 of the outer lens1813 by substantially 90 degrees to direct the optical rays passingalong the optical axis onto the sensor 1853.

In the FIG. 18 embodiment each of the optical chain modules (1890, 1891,1892, 1893, 1894) includes, in addition to an outer lens (1813, 1815,1817, 1819, 1821) a mirror or other device, e.g., prism, (1823, 1825,1827, 1829, 1831), respectively, for changing the angle of lightreceived via the corresponding outer lens (1813, 1815, 1817, 1819,1821), respectively. Additionally, as in some of the previouslydescribed embodiments such as the FIG. 1A, 1B, 1C, 1D, and 3embodiments, each optical chain module (1890, 1891, 1892, 1893, 1894),includes a filter (1833, 1835, 1837, 1839, 1841), respectively, and aninner lens (1843, 1845, 1847, 1849, 1851), respectively. In additioneach optical chain module (1890, 1891, 1892, 1893, 1894) includes asensor (1853, 1855, 1857, 1859, 1861), respectively. For example, thefirst optical chain module (OCM 1 1890) include outer lens L1 1813,mirror 1823, filter 1833, inner lens L2 1843 and sensor 1853.

Filters 1833, 1835, 1837, 1839, and 1841 are mounted on a movablecylinder 1875 represented as a circle shown using small dashed lines.The cylinder 1875 may be rotated and/or moved forward or backwardallowing lenses and/or filters on the cylinder to be easily replacedwith other lenses, filter, or holes mounted on the cylinder 1875. Whilein the FIG. 18 example, an exit hole is provided to allow light to exitcylinder 1875 after passing through one of the filters 1833, 1835, 1837,1839, or 1841 it should be appreciated that rather than an exit holeanother lens or filter may be mounted on the cylinder 1875 allowing twoopportunities for the light to be filtered and/or passed through a lensas is passes through the cylinder 1875. Thus, in at least someembodiments a second filter or lens which is not shown in FIG. 18 forsimplicity is included at the exit point for the light as it passesthrough cylinder 1804. Inner lenses are mounted on cylinder 1885 whichis actually closer to the outside sidewalls of the camera device 1801than the filters mounted on cylinder 1875. Given the large diameter ofmovable cylinder 1885 and the relatively small diameter of the lightbeam as it nears the sensor, it should be appreciated that a largenumber of alternative filters, lenses and/or holes can be mounded oncylinder 1885. As with cylinder 1875 the light can be filtered and/orprocessed by a lens as it enters and leaves cylinder 1885 prior toreaching the sensor of the corresponding optical chain.

In some embodiments lenses mounted on a moveable platter positionedbetween the outer lens platter 1803 and mirrors which may, and in someembodiments are, also mounted on a platter are used to supportautofocus. In such an embodiment the lens platter between the outer lensplatter and mirror platter is moved in or out to perform focusoperations for each of the optical chain modules in parallel. In anotherembodiment, different sets of lens are mounted on the drum 1885 or 1875with different lens sets being mounted with a different offset distancefrom the surface of the drum. By switching between the different sets oflenses by rotating the drum on which the different lens sets aremounted, focusing between different predetermined focus set points can,and in some embodiments is achieved, by simply rotating the drum onwhich the lens sets, corresponding to the different focal distance setpoints, are mounted.

Notably, the FIG. 18 embodiment, by changing the direction of lightthrough the use of mirrors, prisms and/or other devices allows for thelength of the individual optical chains to be longer than the cameradevice is thick. That is, the side to side length of the camera device1801 can be used in combination with a portion of the front to backlength to create optical chains having a length longer than the depth ofthe camera device 1801. The longer optical chain length allows for morelenses and/or filters to be used as compared to what may be possiblewith shorter optical chain lengths. Furthermore, the change in thedirection of light allows for the use of cylinders for mounting lenses,filters and/or holes which can be easily interchanged by a simplerotation or axial, e.g., front to back movement, of the cylinder onwhich the lenses, filters and/or holes corresponding to multiple opticalchains are mounted.

In the FIG. 18 embodiment sensors may be fixed and/or mounted on amovable cylinder 1899. Thus, not only can the lenses, filters and/orholes be easily switched, changes between sensors or sets of sensor canbe easily made by rotating the cylinder on which the sensors aremounted. While a single mirror is shown in FIG. 18 in each optical chainmodule, additional mirrors may be used to further extend the length ofthe optical path by reflecting in yet another direction within thehousing of the camera device 1801.

It should be appreciated that the FIG. 18 embodiment allows for acombination of lens, filter, and/or hole mounting platters arrangedparallel with the platter extending left to right within the cameradevice and cylinders arranged so that the top and bottom of the cylinderextend in the front to back direction with respect to the camera body,e.g., with the front of the camera being shown in FIG. 18. Cylinders maybe mounted inside of one another providing a large number ofopportunities to mount lens, filters and/or holes along the opticalpaths of each optical chain module and allowing for a large number ofpossible filter/lens/sensor combinations to be supported, e.g., byallowing for different combinations of cylinder positions for differentmodes of operation.

While changing sensors mounted on a cylinder can be achieved by rotatinga cylinder, in the earlier embodiments in which sensors may be mountedon platters, sensors may be changed by rotating or otherwise moving aplatter on which the sensors are mounted.

Note that in the FIG. 18 embodiment the outer lenses (1813, 1815, 1817,1819, 1821, of the optical chain modules (1890, 1891, 1892, 1893, 1894),respectively, are mounted near the center of the front of the cameradevice 1801 as shown, e.g., forming a generally circular pattern ofouter lenses 1813, 1815, 1817, 1819, 1821.

In camera device 1801 the optical axes (1805, 1806, 1807, 1808, 1809) oflenses (1813, 1815, 1817, 1819, 1821) said optical chain modules (1890,1891, 1892, 1893, 1894) are parallel to each other but at least twomirrors (1823, 1825) corresponding to different optical chains (1890,1891) are not parallel. The light rays of at least two different opticalchains (1890, 1891) cross prior to reaching the sensor (1853, 1855) towhich the rays of said at least two different optical chain modules(1890, 1891) correspond.

In various embodiments, each optical chain module (1890, 1891, 1892,1893, 1894) includes an image deflection element which includes at leastone mirror positioned at 45 degree to said optical axis (1890, 1891,1892, 1893, 1894) of said lens of the optical chain module. For example,with regard to optical chain module 1 1890, in one embodiments, theimage deflection element 1823 is a mirror positioned at 45 degree to theoptical axis 1805 of lens 1813.

In some embodiments, an image deflection element, e.g., image deflectionelement 1823 includes a prism. In some embodiments, an image deflectionelement includes multiple mirrors. In some embodiments, an imagedeflection element includes a combination including at least one mirrorand at least one prism.

FIG. 19 is similar to the FIG. 18 embodiment in that it illustratesanother camera device 1901 including a plurality of optical chainmodules which include mirrors or another device for changing the angleof light entering the optical chain module and thereby allowing at leasta portion of the optical chain module to extend in a direction, e.g., aperpendicular direction, which is not a straight front to back directionwith respect to the camera device. FIG. 19 illustrates another exemplarycamera device 1901 including a plurality of first through fifth opticalchain modules (1990, 1991, 1992, 1993, 1994) each of which includes anouter lens (1913, 1915, 1917, 1919, 1921), respectively, represented asa circle on the outer lens platter 1903. FIG. 19 differs from the FIG.18 embodiment in that the outer lenses (1913, 1915, 1917, 1919, 1921) ofthe first through fifth optical chain modules (1990, 1991, 1992, 1993,1994) are positioned near the perimeter of the face of the camera device1901. This allows for the length of the optical chain module to belonger than the length of the optical chains shown in FIG. 18. FIG. 19shows outer and inner cylinders, also some times referred to as drums,1975, 1985, upon which filters, lenses and holes can and in variousembodiments are mounted as discussed with regard to the FIG. 18embodiment. Thus cylinders 1975 and 1985 server the same or similarpurpose served by cylinders 1875, 1885, respectively. It should beappreciated that in some embodiments the FIG. 19 embodiment includesfilters and lenses mounted on the inner and outer cylinders in the sameor similar manner as filters and lenses are mounted on the cylinders1875, 1885 shown in FIG. 18.

Elements of the FIG. 19 embodiment which are the same or similar to theelements of the FIG. 18 embodiment are identified beginning with “19”instead of “18” and for the sake of brevity will not be described againin detail. For example element 1961 is used to refer to the sensor forthe optical chain module 1994 which includes outer lens 1921,mirror/light redirection device 1931, filter 1941 and inner lens 1951.The cylinder 1975 is used to mount the filters while cylinder 1985 isused to mount the inner lenses.

Each outer lens (1913, 1915, 1917, 1919, 1921) has an optical axis(1905, 1906, 1907, 1908, 1909), respectively. The optical axis (1905,1906, 1907, 1908, 1909) is represented by an X, indicating that the axisgoes down into the lens (1913, 1915, 1917, 1919, 1921). The optical axis(1905, 1906, 1907, 1908, 1909), are parallel to each other.

The camera devices 1801 and 1901 may, and in some embodiments do,include a processor, display and/or other components of the cameradevice shown in FIG. 1A but such elements are not explicitly shown inthe FIGS. 18 and 19 embodiments to avoid complicating the figures andbeing repetitive.

Various functions of the present invention may be and are implemented asmodules in some embodiments. The assembly of modules 1300 shown in FIG.13 illustrates an exemplary assembly of modules, e.g., software orhardware modules, that may be and are used for performing variousfunctions of a image processing system or apparatus used to processimages in accordance with embodiments of the present invention. When themodules identified in FIG. 13 are implemented as software modules theymay be, and in some embodiments of the present invention are, stored inmemory 108 of FIG. 1A in the section of memory identified as assembly ofmodules 118. These modules may be implemented instead as hardwaremodules, e.g., circuits.

The ideas and concepts described with regard to various embodiments suchas those shown in FIG. 19 can be extended so that the input sensors canbe located in a plane, e.g., at the back of the camera device and/or atthe front of the camera device. In some such embodiments the sensors ofmultiple optical chains are mounted on a flat printed circuit board orbackplane device. The printed circuit board, e.g. backplane, can bemounted or coupled to horizontal or vertical actuators which can bemoved in response to detected camera motion, e.g., as part of a shakecompensation process which will be discussed further below. In some suchembodiments, pairs of light diverting devices, e.g., mirrors, are usedto direct the light so that at least a portion of each optical chainextends perpendicular or generally perpendicular to the input and/orsensor plane. Such embodiments allow for relatively long optical pathswhich take advantage of the width of the camera by using mirrors orother light diverting devices to alter the path of light passing throughan optical chain so that at least a portion of the light path extends ina direction perpendicular or generally perpendicular to the front of thecamera device. The use of mirrors or other light diverting devicesallows the sensors to be located on a plane at the rear or front of thecamera device as will now be discussed in detail.

While the invention has been explained using convex lenses in many ofthe diagrams, it should be appreciated that any of a wide variety ofdifferent types of lenses may be used in the optical chain modulesincluding, e.g., convex, concave, and meniscus lenses. In addition,while lenses and filters have been described as separate elements,lenses and filters may be combined and used. For example, a color lensmay, and in some embodiments is, used to both filter light and alter thelights path. Furthermore, while many of the embodiments have beendescribed with a color filter preceding the image sensor of an opticalchain or as using an image sensor with an integrated color filter, e.g.,a Bayer pattern filter, it should be appreciated that use of colorfilters and/or sensors with color filters is not required and in someembodiments one or more optical chain modules are used which do notinclude a color filter and also do not use a sensor with a color filter.Thus, in some embodiments one or more optical chain modules which sensea wide spectrum of color light are used. Such optical chain modules areparticularly well suited for generating black and white images.

In various embodiments image processing is used to simulate a widevariety of user selectable lens bokehs or blurs in the combined imagewith regard to image portions which are out of focus. Thus, whilemultiple lenses are used to capture the light used to generate acombined image, the image quality is not limited to that of anindividual one of the lenses and a variety of bokehs can be achieveddepending on the particular bokeh desired for the combined image beinggenerated. In some embodiments, multiple combined images with differentsimulated bokehs are generated using post image capture processing withthe user being provided the opportunity to save one or more of thegenerated combined images for subsequent viewing and/or printing. Thus,in at least some embodiments a physical result, e.g., a printed versionof one or more combined images is produced. In many if not all casesimages representing real world objects and/or scenes which were capturedby one or more of the optical chain modules of the camera device used totake the picture are preserved in digital form on a computer readablemedium, e.g., RAM or other memory device and/or stored in the form of aprinted image on paper or on another printable medium.

While explained in the context of still image capture, it should beappreciated that the camera device and optical chain modules of thepresent invention can be used to capture video as well. In someembodiments a video sequence is captured and the user can select anobject in the video sequence, e.g., shown in a frame of a sequence, as afocus area, and then the camera device capture one or more images usingthe optical chain modules. The images may, and in some embodiments are,combined to generate one or more images, e.g., frames. A sequence ofcombined images, e.g., frames may and in some embodiments is generated,e.g., with some or all individual frames corresponding to multipleimages captured at the same time but with different frames correspondingto images captured at different times.

While different optical chain modules are controlled to use differentexposure times in some embodiments to capture different amounts of lightwith the captured images being subsequently combined to produce an imagewith a greater dynamic range than might be achieved using a singleexposure time, the same or similar effects can and in some embodimentsis achieved through the use of different filters on different opticalchains which have the same exposure time. For example, by using the sameexposure time but different filters, the sensors of different opticalchain modules will sense different amounts of light due to the differentfilters which allowing different amounts of light to pass. In one suchembodiment the exposure time of the optical chains is kept the same byat least some filters corresponding to different optical chain modulescorresponding to the same color allow different amounts of light topass. In non-color embodiments neutral filters of different darknesslevels are used in front of sensors which are not color filtered. Insome embodiments the switching to a mode in which filters of differentdarkness levels is achieved by a simple rotation or movement of a filterplatter which moves the desired filters into place in one or moreoptical chain modules. The camera devices of the present inventionsupports multiple modes of operation with switching between panoramicmode in which different areas are captured, e.g., using multiple lensesper area, and a normal mode in which multiple lens pointed samedirection are used to capture the same scene. Different exposure modesand filter modes may also be supported and switched between, e.g., basedon user input.

Various functions of the present invention may be and are implemented asmodules in some embodiments. The assembly of modules 1700 shown in FIG.13 illustrates an exemplary assembly of modules, e.g., software orhardware modules, that may be and are used for performing variousfunctions of a image processing system or apparatus used to processimages in accordance with embodiments of the present invention. When themodules identified in FIG. 13 are implemented as software modules theymay be, and in some embodiments of the present invention are, stored inmemory 108 of FIG. 1A in the section of memory identified as assembly ofmodules 118. These modules may be implemented instead as hardwaremodules, e.g., circuits.

The ideas and concepts described with regard to various embodiments suchas those shown in FIG. 19 can be extended so that the input sensors canbe located in a plane, e.g., at the back of the camera device and/or atthe front of the camera device. In some such embodiments the sensors ofmultiple optical chains are mounted on a flat printed circuit board orbackplane device. The printed circuit board, e.g. backplane, can bemounted or coupled to horizontal or vertical actuators which can bemoved in response to detected camera motion, e.g., as part of a shakecompensation process which will be discussed further below. In some suchembodiments, pairs of light diverting devices, e.g., mirrors, are usedto direct the light so that at least a portion of each optical chainextends perpendicular or generally perpendicular to the input and/orsensor plane. Such embodiments allow for relatively long optical pathswhich take advantage of the width of the camera by using mirrors orother light diverting devices to alter the path of light passing throughan optical chain so that at least a portion of the light path extends ina direction perpendicular or generally perpendicular to the front of thecamera device. The use of mirrors or other light diverting devicesallows the sensors to be located on a plane at the rear or front of thecamera device as will now be discussed in detail.

In the FIGS. 20 and 21 embodiments two or more deflection elements areused in each optical chain. Mirrors are exemplary deflection elementsthat may and sometimes are used in the FIGS. 20 and 21 embodiments.Thus, at least in some embodiments each optical chain includes multipledeflection elements in the form of mirrors. In FIGS. 20 and 21embodiments two deflection elements are used in each optical chain witheach deflection element, e.g., mirror, deflecting the light 90 degrees.

FIG. 20 illustrates an exemplary diagram of a camera device 2000implemented in accordance with one exemplary embodiment of theinvention. The FIG. 20 diagram is intended for explanation purposes tofacilitate an understanding of various features and thus is not aprecise view of the camera device as perceived from the top but afunctional diagram of the elements from a top view perspective which isintended to convey various aspects of the optical chain configurationsused in the device 2000. The top portion of FIG. 20 corresponds to thefront of the camera device 2000 while the bottom portion corresponds tothe back of the camera device 2000. The body 2001 of the camera extendsfrom left to right with the lens and/or openings 2002, 2004, 20006corresponding to multiple optical chains being mounted in front of thecamera device 2000. A LCD or other display (not shown) may and in someembodiments is, located at the rear of the camera device 2000.

In the camera device 2000 includes a plurality of lens or openings L1through LZ 2002, 2004, 2006 each corresponding to a different one of Zoptical chains. Note that in FIG. 20 the lenses 2002, 2004 and 2006 areloaded in a plane represented by dashed line 2012 which extends downtowards the bottom of the camera device 2000 which is not visible in theFIG. 20 diagram. The lenses 2002, 2004, and 2006 may be arranged in acircular or other pattern on the front of the camera device 2002. Eachoptical chain in the FIG. 20 embodiment includes multiple mirrors orother light redirecting devices and a sensor positioned at the end ofthe optical chain. For example, optical chain 1 includes lens 2002,first mirror 2022, second mirror 2024 and sensor 2038. Optical chain Zincludes lens LZ 2006, first mirror 2028, second mirror 2026 and sensorZ 2034. It should be appreciated that mirrors of the first and secondoptical chains are located around the cylinder 2020 on which one or morelenses or filters may be mounted as discussed with regard to the otherembodiments. The mirrors may be arranged in a plane positioned parallelto the input plane 2012 with the light of the different optical chainspassing each other, e.g., crossing, within the cylinder 2020. While asingle cylinder 2020 is shown in FIG. 20, multiple cylinders, lensesand/or filters may, and in some embodiments are, used as discussed withregard to the other embodiments. Note that in the FIG. 20 embodiment themirrors (2022, 2024), (2028, 2026) redirect the light passing throughthe optical chain to which the mirrors correspond so that at least aportion of the optical path of the optical chain extends perpendicularor generally perpendicular to the input direction in which the inputlenses L1, L2, LZ face and parallel to the input plane 2012. The inputplane may be implemented as a mounting device, e.g., circuit board, uponone or more input lenses or openings L1, L2, LZ are mounted or includedin. This allows the optical chain to take advantage of the left to rightwidth of a camera permitting an overall optical chain length than wouldbe possible if the optical chain was limited to the front to back depthof camera device 2000. This allows for thin cameras with relatively longoptical chains. Notably, the use of two 45 degree mirrors 2022, 2024allows the sensors of the optical chain to be mounted in a backplane2030 with the sensors being arranged on the backplane 2030 in a planewhich is parallel to the input plane 2012. The ability to mount thesensors on a single backplane allows for the simple movement of thesensors as an assembly maintaining the relative position of the sensors2034, 2038 to one another on the backplane 2030 even if the backplane ismoved. The cylinder and mirrors may, but need not be, mounted in amanner so that they will move with the backplane 2030 maintaining thealignment of the optical chains to one another as the backplane 2030 ismoved, e.g., up or down or left to right in the camera body 2000. Thus,in some embodiments the backplane 2030 and sensors 2034, 2038 can bemoved in unison, e.g., by applying a force to the backplane 2030 toinduce motion as may be desired.

In one embodiment, motion sensors 2040 are included in the camera device2000. The motion sensors 2040 may be accelerometers and/or gyroscopesused to detect motion along one or more axis of the camera. In oneparticular embodiment a shake compensation module 2042 is included inthe camera device 2000. The shake compensation module 2042 receivesoutput from the motion sensors 2040 and detects camera movement, e.g.,movement indicative of un-intentional shaking as is common in the caseof hand held cameras. The shake compensation control module is coupledto a horizontal actuator 2032 and a vertical actuator 2036 which are incontact with the backplane 2030 which may be a circuit board. Thevertical actuator 2036 is shown in dashed lines since it is positionedbelow backplane 2030 and would not be visible from the top. The verticalactuator 2036 can be used to move the backplane 2030, e.g. circuitboard, up or down while actuator 2032 can be used to move the backplane2030 left or right. In at least one embodiment backplane 2030 is mountedin a manner that allows motion left and right, up and down, but whichmaintains its parallel relationship to the input plane 2012. In someembodiments backplane 2030 is mounted in a slot which is part of thehousing of the camera device 2000. The actuators 2032, 3036 may bemotorized or implemented using elements which expand or contract when avoltage is supplied. The shake compensation control module 2042 controlsthe supply of power and/or control signals to actuators 2032, 2036 whichinduces motion of the backplane 2030 and sensors mounted thereon whichis intended to counteract the shaking. The motion of the backplane 2030is normally not detectable to the holder of the camera but can reducethe distorting in the captured images induced by shaking of the camerahousing in which the various elements of the camera are mounted. Thelenses and/or openings 2002, 2004, 2006 may not distort or focus theincoming light and may remain fixed while one or more of the otherelements of the optical chains move, e.g., to compensate for shakingand/or changes the lenses on the cylinder or drum 2020 through whichlight will pass.

The FIG. 20 embodiment is particular well suited for embodiments whereit is desirable from a manufacturing standpoint and/or shakecompensation standpoint to mount the sensors 2034, 2038 on backplanessuch as printed circuit boards or other relatively flat mounting deviceswhether they be out of metal, plastic, another material or a combinationof materials. It should be appreciated that the camera device 2000, aswell as the camera device 2100 shown in FIG. 21 may include the elementsof the camera device 100 shown in FIG. 1A in addition to those shown inFIGS. 20 and 21 but that such elements are omitted to facilitate anunderstanding of the elements and configuration which is explained usingFIGS. 20 and 21.

FIG. 21 illustrates an additional exemplary camera device 2100 in whichmirrors (2122, 2124), (2128, 2126) and/or other light redirectingelements are used to alter the path of light in the optical chains sothat the input light input lenses and/or opens can be arranged in one ormore planes at the front of the camera where the lens and/or openingsthrough which light enters the optical chains are also located. Elementsin FIG. 21 which are the same or similar the elements of FIG. 20 arenumbered using similar numbers but starting with the first two digits 21instead of 20. Such similar elements will not be described again expectto point out some of the differences between the FIG. 21 and FIG. 20configurations. One of the differences between the devices 2100 and 2000is that in the camera device 2100 both the sensors 2134, 2138 andexternal lenses/openings of the optical chains are located in the frontof the camera. This is made possible by having the second mirror 2124 or2126 direct light to the front of the camera rather than the back of thecamera.

Numerous variations on the designs shown in FIGS. 20 and 21 arepossible. Significantly, the methods and apparatus of the presentinvention allow for sensors to be arranged parallel to or on anyinternal wall of a camera device while still allowing for a cameradevice to include multiple optical chains in a relatively thin camera.By configuring the sensors parallel to the front or rear walls of thecamera rather than the side walls, the sensors and/or lens can be spreadout and occupy a greater surface area than might be possible if thecamera sensors were restricted to the sidewalls or some otherarrangement. Notably many of the embodiments are well suited forallowing a LCD or other display to be placed at the back of the camerafacing out without the display panel significantly interfering with theoverall length of the individual optical chain modules included in thecamera.

While the invention has been explained using convex lenses in many ofthe diagrams, it should be appreciated that any of a wide variety ofdifferent types of lenses may be used in the optical chain modulesincluding, e.g., convex, concave, and meniscus lenses. In addition,while lenses and filters have been described as separate elements,lenses and filters may be combined and used. For example, a color lensmay, and in some embodiments is, used to both filter light and alter thelights path. Furthermore, while many of the embodiments have beendescribed with a color filter preceding the image sensor of an opticalchain or as using an image sensor with an integrated color filter, e.g.,a Bayer pattern filter, it should be appreciated that use of colorfilters and/or sensors with color filters is not required and in someembodiments one or more optical chain modules are used which do notinclude a color filter and also do not use a sensor with a color filter.Thus, in some embodiments one or more optical chain modules which sensea wide spectrum of color light are used. Such optical chain modules areparticularly well suited for generating black and white images.

In various embodiments image processing is used to simulate a widevariety of user selectable lens bokehs or blurs in the combined imagewith regard to image portions which are out of focus. Thus, whilemultiple lenses are used to capture the light used to generate acombined image, the image quality is not limited to that of anindividual one of the lenses and a variety of bokehs can be achieveddepending on the particular bokeh desired for the combined image beinggenerated. In some embodiments, multiple combined images with differentsimulated bokehs are generated using post image capture processing withthe user being provided the opportunity to save one or more of thegenerated combined images for subsequent viewing and/or printing. Thus,in at least some embodiments a physical result, e.g., a printed versionof one or more combined images is produced. In many if not all casesimages representing real world objects and/or scenes which were capturedby one or more of the optical chain modules of the camera device used totake the picture are preserved in digital form on a computer readablemedium, e.g., RAM or other memory device and/or stored in the form of aprinted image on paper or on another printable medium.

While explained in the context of still image capture, it should beappreciated that the camera device and optical chain modules of thepresent invention can be used to capture video as well. In someembodiments a video sequence is captured and the user can select anobject in the video sequence, e.g., shown in a frame of a sequence, as afocus area, and then the camera device capture one or more images usingthe optical chain modules. The images may, and in some embodiments are,combined to generate one or more images, e.g., frames. A sequence ofcombined images, e.g., frames may and in some embodiments is generated,e.g., with some or all individual frames corresponding to multipleimages captured at the same time but with different frames correspondingto images captured at different times.

While different optical chain modules are controlled to use differentexposure times in some embodiments to capture different amounts of lightwith the captured images being subsequently combined to produce an imagewith a greater dynamic range than might be achieved using a singleexposure time, the same or similar effects can and in some embodimentsis achieved through the use of different filters on different opticalchains which have the same exposure time. For example, by using the sameexposure time but different filters, the sensors of different opticalchain modules will sense different amounts of light due to the differentfilters which allowing different amounts of light to pass. In one suchembodiment the exposure time of the optical chains is kept the same byat least some filters corresponding to different optical chain modulescorresponding to the same color allow different amounts of light topass. In non-color embodiments neutral filters of different darknesslevels are used in front of sensors which are not color filtered. Insome embodiments the switching to a mode in which filters of differentdarkness levels is achieved by a simple rotation or movement of a filterplatter which moves the desired filters into place in one or moreoptical chain modules. The camera devices of the present inventionsupports multiple modes of operation with switching between panoramicmode in which different areas are captured, e.g., using multiple lensesper area, and a normal mode in which multiple lens pointed samedirection are used to capture the same scene. Different exposure modesand filter modes may also be supported and switched between, e.g., basedon user input.

Numerous additional variations and combinations are possible whileremaining within the scope of the invention.

FIG. 22, comprising the combination of FIG. 22A and FIG. 22B, is aflowchart 2200 of an exemplary method of operating a camera deviceincluding a plurality of optical chains, said plurality of opticalchains including at least a first group of optical chains and a secondgroup of optical chains in accordance with an exemplary embodiment. Insome embodiments, the exemplary method of flowchart 2200 uses multiplegroups of lenses to support continuous zooming with a combination ofdigital zoom and discrete lens focal length changes. The method of FIG.22 can be implemented by any one of the camera devices described in thepresent application including, for example, camera device 100 of FIG.1A.

As should be appreciated, focal length is an indication of the opticaldistance from the point where light rays converge to form a sharp imageof an object to the digital sensor, e.g., at the focal plane of theoptical chain to which the focal length relates. The focal lengthprovide information on the angel of view and thus how much of a scenewill be captured by a sensor of an optical chain as well themagnification, e.g., how large an individual element will appear whensensed by the sensor. The longer the focal length, the narrower theangle of view and the higher the magnification. The shorter the focallength the wider the angle of view and the lower the magnification willbe. Thus, by using different focal lengths, e.g., by changing or morelenses or lens positions in an optical chain, an optical chain canprovide different amounts of magnification and capture different amountof an image area as a function of the focal length. While lenses can beused to provide optical chains with actual focal lengths, digital imageprocessing can be performed to enlarge an image and thus produce imageswith simulated focal lengths greater than the actual focal lengths.

While mechanical systems can be devices to move lenses and provide acontinuous change in actual focal length, mechanical systems to supportsmooth changes in focal length can be costly to implement. It is mucheasier to make changes between focal lengths in discrete, e.g., fixedsizes amounts or units, e.g., by changing lenses. However, from a userperspective it can be desirable to support a smooth or continuous zoomfunction.

In accordance with various embodiments, a smooth zoom is achieved by acombination of discrete focal length changes in combination withsimulated focal length changes between the discrete changes. To supportthe continuous zoom, in some embodiments multiple optical chain modulesare separated into groups with the focal length of one group beingchanged by a discrete amount, e.g., by a lens change, while the focallength and image capture capabilities of the other group of opticalchains continues to be used.

By using at least two groups of optical chains that discretely changetheir focal lengths at different times in combination with imageprocessing being used to simulate changes in focal length, e.g., byperforming an enlargement operation electronically on the image data, auser is provided with the appearance of a smooth zoom operation eventhough the optical chains have their focal lengths switched by discreteamounts from time to time as needed to support the desired zoom in orzoom out operation.

In some embodiments, each of the steps of flowchart 2200 are implementedby a camera device including groups of optical chains. In someembodiments, the camera device including groups of optical chains is acell phone or other portable camera device, e.g., an electronic tablet,electronic pad, webcam device, surveillance device, etc.

In other embodiments, the image combining steps are implemented by adevice, e.g., a computer system, external to the camera device, and theother steps are implemented by the exemplary camera device includinggroups of optical chains.

Operation starts in step 2202 and proceeds to step 2203. In step 2203values are initialized for: the first focal length, the second focallength, the fourth simulated focal length, the first simulated focallength, the second simulated focal length, third simulated focal lengthand the fifth simulated focal length. In one example, the first focallength f1 is set to 30; the second focal length f2 is set to 100; thefourth simulated focal length S4 is set to 30; the first simulated focallength S1 is set to 100; the second simulated focal length S2 is set to105; the third simulated focal length S3 is set to 110; and the fifthsimulated focal length S5 is set to 400. This set-up allows the cameradevice to implement a zoom in operation.

Operation proceeds from step 2203 to step 2204. In step 2204 aplurality, e.g., N, of optical chains are provided, said plurality ofoptical chains including at least a first group of optical chains and asecond group of optical chains. In some embodiments, providing aplurality of optical chains included providing a camera device includingthe plurality of optical chains. In some embodiments, N is at least 6,and each of the first and second groups of optical chains includes atleast 3 optical chains. In some embodiments, N is at least 8, and eachof the first and second groups of optical chains includes at least 4optical chains. Operation proceeds from step 2204 to step 2206. In step2206 image data is captured using at least some of said plurality ofoptical chains during a third period of time. Operation proceeds fromstep 2206 to step 2208.

In step 2208 the image data captured using at least some of saidplurality of optical chains is combined to form a video image streamincluding composite image generated by combining images from multipleoptical chains, said multiple optical chains including at least oneoptical chain from each of the first and second groups of opticalchains. Step 2208 includes step 2210 in which image data captured duringsaid third time period by at least said first and second groups ofoptical chains is combined, said combining including performing a thirddigital zoom operation which simulates a continuous zoom during saidthird period of time starting with a fourth simulated focal length andending with a first simulated focal length. Operation proceeds from step2208 to steps 2212, 2214, and 2216. Steps 2212, 2214 and 2216 may be,and sometimes are, performed in parallel.

In step 2212 the first group of optical chains is transitioned from afirst focal length to a second focal length during a first period oftime. In step 2214 images are captured using the second group of opticalchains during the first period of time. In step 2216 image data capturedby the first group of optical chains during the first period of time isexcluded from use in generating a composite image or image data is notcaptured from the first group of optical chains during the first periodof time. Operation proceeds from steps 2212, 2214 and 2216 to step 2218.

In step 2218, image data captured during the first period of time by atleast the second group of optical chains is combined, without using anyimage data from the first group of optical chains, to form a video imagestream, said combining including performing a first digital zoomoperation which simulates a continuous zoom during the first period oftime. Operation proceeds from step 2218, via connecting node A 2219 tosteps 2220, 2222, and 2224. Steps 2220, 2222 and 2224 may be, andsometimes are, performed in parallel.

In step 2220 the second group of optical chains is transitioned fromsaid first focal length to said second focal length during a secondperiod of time. In some embodiments, at least one of the first andsecond groups of optical chains include optical chains with controllablefocal lengths which can be changed at points in time. In step 2222images from the first group of optical chains are captured during thesecond period of time, optical chains in said first group of opticalchains have a different focal length than optical chains in said secondgroup of optical chains during a least a portion of the second period oftime. In step 2224 image data captured by the second group of opticalchains during the second period of time is excluded from use ingenerating a composite image or image data is not captured from thesecond group of optical chains during the second period of time.Operation proceeds from steps 2220, 2222 and 2224 to step 2226.

In step 2226, image data captured during the second period of time by atleast the first group of optical chains is combined, without using anyimage data from the second group of optical chains, to form a videoimage stream, said combining including performing a second digital zoomoperation which simulates a continuous zoom during the second period oftime. Operation proceeds from step 2226 to step 2228.

In step 2228 image data is captured using at least some of saidplurality of optical chains during a fourth period of time. Operationproceeds from step 2228 to step 2230. In step 2230 the image datacaptured using at least some of said plurality of optical chains duringthe fourth time period is combined to form a video image streamincluding composite images generated by combining image from multipleoptical chains including at least one optical chain from each of thefirst and second groups of optical chains. Step 2230 includes step 2232.In step 2232 image data captured during said fourth time period by atleast said first and second groups of optical chains is combined, saidcombining including performing a fourth digital zoom operation whichsimulates a continuous zoom during said fourth period of time startingwith the third simulated focal length and end with said fifth simulatedfocal length.

Operation proceeds from step 2230 to step 2234. In step 2234, new valuesare set for: the first focal length, the second focal length, the fourthsimulated focal length the first simulated focal length, the secondsimulated focal length, and the fifth simulated focal length. In oneexample, the first focal length f1 is set to 100; the second focallength f2 is set to 30; the fourth simulated focal length S4 is set to400; the first simulated focal length S1 is set to 110; the secondsimulated focal length S2 is set to 105; the third simulated focallength S3 is set to 100; and the fifth simulated focal length S5 is setto 30. This set-up allows the camera device to implement a zoom outoperation. Operation proceeds from step 2234, via connecting node B 2235to step 2206.

In some embodiments, the first digital zoom during the first period oftime begins with a first simulated focal length and ends with a secondsimulated focal length. In some such embodiments, the first simulatedfocal length is equal to or larger than the first focal length. In somesuch embodiments, the first digital zoom is accomplished by taking theentire image or performing an image cropping operation to select aportion of the image.

In various embodiments, the second simulated focal length is equal to orlarger than the second focal length. In some such embodiments, saidfirst digital zoom is accomplished by taking the entire image orperforming an image cropping operation to select a portion of saidimage, e.g., simulating a larger focal length than the lens focallength.

In some embodiments, the digital zoom during the second period of timebegins with the second simulated focal length and ends with a thirdsimulated focal length. In some such embodiments, the second simulatedfocal length is equal to or larger than the second focal length. In somesuch embodiments, the second digital zoom is accomplished by taking theentire images or performing image cropping operations to select aportion of said images.

In various embodiments, the third simulated focal length is equal to orlarger than the second focal length. In some such embodiments, thesecond digital zoom is accomplished by taking the entire image orperforming an image cropping operation to select a portion of saidimage, e.g., simulating a larger focal length.

In various embodiments, the fourth simulated focal length is greaterthan or equal to the first focal length. In some embodiments, the fifthsimulated focal length is greater than or equal to second focal length.

In some embodiments, the fourth simulated focal length is less than thefirst simulated focal length; the first simulated focal length is lessthan the second simulated focal length; the second simulated focallength is less than the third simulated focal length; the thirdsimulated focal length is less than the fifth simulated focal length;and the first focal length is less than the second focal length. In somesuch embodiments, this supports a zoom in operation.

In some embodiments, the fourth simulated focal length is greater thanthe first simulated focal length; the first simulated focal length isgreater than the second simulated focal length, the second simulatedfocal length is greater than the third simulated focal length; the thirdsimulated focal length is greater than the fifth simulated focal length;and the first focal length is greater than the second focal length. Insome such embodiments, this supports a zoom out operation.

In various embodiments, the rate of change of the In (natural log) ofthe simulated focal length during the first, second, third, and fourthtime periods is the same and is constant. This approach can be, and insome embodiments is, used to result in a continuous zoom at a fixedrate. The rate of change of the log of the simulated focal length wouldalso be constant and is related to the natural log by a constant.

In some embodiments, the focal length of optical chains in the secondgroup of optical chains remains fixed and does not change during thefirst period of time. In some such embodiments, the focal length ofoptical chains in the first group of optical chains remains fixed anddoes not change during the second period of time.

In some embodiments, the first group of optical chains includes opticalchains with fixed focal length which do not change with respect to time.In some such embodiments, the second group of optical chains includesoptical chins with fixed focal length which do not change with respectto time.

In some embodiments, at least one of the first and second groups ofoptical chains include optical chains with controllable focal lengthswhich can be changed at points in time. In some such embodiments, boththe first and second groups of optical chains include optical chainswith controllable focal lengths which can be changed at points in time.

FIG. 23 is a drawing 2300 illustrating an example in which the method offlowchart 2200 is implemented corresponding to a zoom in operation,e.g., an operation where an item in an image will appear to becomelarger as the zoom increases along with the actual and/or simulatedfocal length of the optical chains being used. In the FIG. 23 example,the focal lengths, both actual and simulated, are in units of mm howeverother units may be used. In this example, there is a first fixed focallength, f1=30 mm, and a second fixed focal length, f2=100 mm. Horizontalaxis 2302 represents time. In this example, there is a third timeinterval T3 2304, which is followed by a first time interval T1 2306.The first time interval T1 2306 is followed by a second time interval T22308, and the second time interval is followed by a fourth time intervalT4 2310.

At the start of the third time interval T3, the camera device has afourth simulated focal length S4=30, as indicated by box 2312. At theend of the third time interval T3 and start of the first time intervalT1, the camera device has a first simulated focal length S1=100, asindicated by box 2314. At the end of the first time interval T1 andstart of the second time interval T2, the camera device has a secondsimulated focal length S2=105, as indicated by box 2316. At the end ofthe second time interval T2 and start of the third time interval T3, thecamera device has a third simulated focal length S3=110, as indicated bybox 2318. At the end of the fourth time interval T4 the camera devicehas a fifth simulated focal length S5=400, as indicated by box 2320.

During the third time interval T3, the first group of optical chains isat a first fixed focal length, f1=30, as indicated by box 2322. Duringthe third time interval T3, the second group of optical chains is at thefirst fixed focal length, f1=30, as indicated by box 2324. During thethird time interval T3, combined image data from both the first andsecond groups of optical chains is used to generate third digital zoomimages, as indicated by box 2326.

During the first time interval T1, the first group of optical chains isbeing transitioned from the first fixed focal length to the second fixedfocal length, as indicated by box 2328. During the first time intervalT1, the second group of optical chains is at the first fixed focallength, f1=30, as indicated by box 2330. During the first time intervalT1, combined image data from the second group of optical chains is usedto generate first digital zoom images, as indicated by box 2332.

During the second time interval T2, the first group of optical chains isat the second fixed focal length, f2=100, as indicated by box 2334.During the second time interval T2, the second group of optical chainsis being transitioned from the first fixed focal length to the secondfixed focal length, as indicated by box 2336. During the second timeinterval T2, combined image data from the first group of optical chainsis used to generate second digital zoom images, as indicated by box2338.

During the fourth time interval T4, the first group of optical chains isat a second fixed focal length, f2=100, as indicated by box 2340. Duringthe fourth time interval T4, the second group of optical chains is atthe second fixed focal length, f1=100, as indicated by box 2342. Duringthe fourth time interval T4, combined image data from both the first andsecond groups of optical chains is used to generate fourth digital zoomimages, as indicated by box 2344.

FIG. 24 is a drawing 2400 illustrating an example in which the method offlowchart 2200 is implemented corresponding to a zoom out operation. Inthis example, there is a first fixed focal length, f1=100, and a secondfixed focal length, f2=30. Horizontal axis 2402 represents time. In thisexample, there is a third time interval T3 2404, which is followed by afirst time interval T1 2406. The first time interval T1 2406 is followedby a second time interval T2 2408, and the second time interval T2 2408is followed by a fourth time interval T4 2410.

At the start of the third time interval T3, the camera device has afourth simulated focal length S4=400, as indicated by box 2412. At theend of the third time interval T3 and start of the first time intervalT1, the camera device has a first simulated focal length S1=110, asindicated by box 2414. At the end of the first time interval T1 andstart of the second time interval T2, the camera device has a secondsimulated focal length S2=105, as indicated by box 2416. At the end ofthe second time interval T2 and start of the third time interval T3, thecamera device has a third simulated focal length S3=100, as indicated bybox 2418. At the end of the fourth time interval T4 the camera devicehas a fifth simulated focal length S5=30, as indicated by box 2420.

During the third time interval T3, the first group of optical chains isat a first fixed focal length, f1=100, as indicated by box 2422. Duringthe third time interval T3, the second group of optical chains is at thefirst fixed focal length, f1=100, as indicated by box 2424. During thethird time interval T3, combined image data from both the first andsecond groups of optical chains is used to generate third digital zoomimages, as indicated by box 2426.

During the first time interval T1, the first group of optical chains isbeing transitioned from the first fixed focal length to the second fixedfocal length, as indicated by box 2428. During the first time intervalT1, the second group of optical chains is at the first fixed focallength, f1=100, as indicated by box 2430. During the first time intervalT1, combined image data from the second group of optical chains is usedto generate first digital zoom images, as indicated by box 2432.

During the second time interval T2, the first group of optical chains isat the second fixed focal length, f2=30, as indicated by box 2434.During the second time interval T2, the second group of optical chainsis being transitioned from the first fixed focal length to the secondfixed focal length, as indicated by box 2436. During the second timeinterval T2, combined image data from the first group of optical chainsis used to generate second digital zoom images, as indicated by box2438.

During the fourth time interval T4, the first group of optical chains isat a second fixed focal length, f2=30, as indicated by box 2440. Duringthe fourth time interval T4, the second group of optical chains is atthe second fixed focal length, f1=30, as indicated by box 2442. Duringthe fourth time interval T4, combined image data from both the first andsecond groups of optical chains is used to generate fourth digital zoomimages, as indicated by box 2444.

FIG. 25, comprising the combination of FIG. 25A and FIG. 25B, is anassembly of modules 2500, which may be included in an exemplary device,e.g., a camera device including groups of optical chains, or exemplarycombination of devices, e.g., a camera device including groups ofoptical chains and a computer device external to the camera device,implementing the method of flowchart 2200 of FIG. 22. Assembly ofmodules 2500 includes Part A 2501 and Part B 2599.

The method of FIG. 22 can be implemented in one embodiment by the camera100. In at least one such embodiment the plurality of optical chainmodules 130 include multiple sets of optical chains with the focallengths of the different sets being capable of being changed, e.g., bychanging one or more lens in the chains in the set. As discussed above,this change can be made in a variety of ways by rotating a drum withdifferent lenses mounted thereon, rotating disc with lenses mountedthereon or using other techniques.

In one embodiment the assembly of modules shown in FIG. 25 is part of orused in place of the assembly of modules 118. Thus, the assembly ofmodules 2500 may be included with the previously described modules inthe assembly 118 or with some or all of the modules shown in FIG. 25being used in place of the modules previously described with regard toassembly 118. The modules in the assembly 2500, when executed by theprocessor 110 control the camera in one embodiment to implement themethod described with regard to FIG. 22. While the modules of FIG. 25may, and in some embodiments are implemented using software, in otherembodiments they are implemented in hardware, e.g., as circuits, whichmay and in some embodiments are included in the camera device 100.

The assembly of modules shown in FIG. 25 include a plurality of modules2503, 2504, 2506, 2508, 2510, 2512, 2514, 2516, 2518, 2520, 2522, 2524,2526, 2528, 2530, 2532, and 2534 for performing the correspondingfunctions of the steps described with regard to the method shown in FIG.22.

Assembly of modules 2500 includes an initialization module 2503configured to initialize values for: first focal length, second focallength, fourth simulated focal length, first simulated focal length,second simulated focal length, and fifth simulated focal length, acontrol module 2504 configured to control a plurality of optical chains,e.g. N optical chains, said plurality of optical chains including atleast a first group of optical chains and a second group of opticalchains. Control module 2504 includes a first group control module 2550configured to control a first group of optical chains and a second groupcontrol module 2552 configured to control a second group of opticalchains. Assembly of modules 2500 further includes a third image capturemodule 2506 configured to capture image data using at least some of saidplurality of optical chains during a third time period, and a third timeperiod image combining module 2508 configured to combine the image datacaptured, e.g., by module 2506, using at least some of the plurality ofoptical chains to form a video image stream including composite imagesgenerated by combining images from multiple optical chains, saidmultiple optical chains including at least one optical chains from eachof the first and second groups of optical chains. Module 2508 includes amodule 2510 configured to combine image data captured during said thirdtime period by at least said first and second groups of optical chins,said combining including performing a third digital zoom operation whichsimulates a continuous zoom during said third period of time startingwith a fourth simulated focal length and ending with said firstsimulated focal length.

Assembly of module 2500 further includes a second transition controlmodule 2512 configured to transition the first group of optical chainsfrom a first focal length to a second focal length during a first periodof time, a first image capture module 2514 configured to capture imagesusing the second group of optical chains during the first period oftime, a first image data exclusion module 2516 configured to excludefrom use in generating a composite image, image data captured by thefirst group of optical chains or not capture image data from the firstgroup of optical chains during the first period of time. Assembly ofmodules 2500 further includes a first time period image combining module2518 configured to combine image data captured during said first periodof time by at least the second group of optical chains, without usingany image data from the first group of optical chains, to form a videoimage stream, said combining including performing a first digital zoomoperation which simulates a continuous zoom during said first period oftime.

Assembly of modules 2500 further includes a first transition controlmodule 2520 configure to transition the second group of optical chainsfrom the first focal length to the second focal length during a secondperiod of time, a second image capture module 2522 configured to captureimages using the first group of optical chins during the second periodof time, and a second image data exclusion module 2524 configured toexclude from use in generating a composite image data captured by thesecond group of optical chains or not capture image data from the secondgroup of optical chains during the second period of time. Assembly ofmodules 2500 further includes a second time period image combiningmodule 2526 configured to combine image data captured during said secondperiod of time by at least the first group of optical chains, withoutusing any image data from the second group of optical chains, to form avideo image stream, said combining including performing a second digitalzoom operation which simulates a continuous zoom during said secondperiod of time.

Assembly of modules 2500 further includes a fourth image capture module2528 configured to capture image data using at least some of saidplurality of optical chains during a fourth period of time, and a fourthtime period image data combining module 2530 configured to combine imagedata captured, e.g., captured by module 2528, using at least some ofsaid plurality of optical chains to form a video image stream includingcomposite images generated by combining images from multiple opticalchins, said multiple optical chains including at least one optical chainfrom each of the first and second groups of optical chains. Module 2530includes a module 2532 configured to combine image data captured duringthe fourth period of time by at least the first and second groups ofoptical chains, said combining including performing a fourth digitalzoom operation which simulates a continuous zoom during the fourthperiod of time starting with the third simulated focal length and endingwith the first simulated focal length.

Assembly of modules 2500 further includes a value setting module 2534configured to set new values for: first focal length, second focallength, fourth simulate focal length, second simulated focal length,third simulated focal length and fifth simulated focal length. Assemblyof modules 2554 further includes an image cropping module 2554configured to crop an image as a part of a zoom operation, a zoomdirection module 2556 configured to select between zoom out and zoom in,e.g., in response to a user input, and to control operation inaccordance with the selection, and a zoom rate control module 2558configured to control the zoom rate in accordance with a predeterminedsetting or in accordance with a user selected rate.

In some embodiments, the focal length of optical chains in the secondgroup of optical chains remains fixed and does not change during saidfirst period of time. In some such embodiments, the focal length ofoptical chains in the first group of optical chains remains fixed anddoes not change during said second period of time.

In some embodiments, the first group of optical chains includes opticalchains with fixed focal lengths which do not change with respect totime. In some such embodiments, the second group of optical chainsincludes optical chains with fixed focal length which do not change withrespect to time.

In various embodiments, at least one of said first and second groups ofoptical chains include optical chains with controllable focal lengthswhich can be changed at points in time. In some such embodiments, bothof the first and second groups of optical chains include optical chainswith controllable focal lengths which can be changed at points in time.

In some embodiments, the first digital zoom during said first period oftime begins with a first simulated focal length and ends with a secondsimulated focal length. In some such embodiments, said first simulatedfocal length is equal to or larger than the first focal length. In someembodiments, the first digital zoom is accomplished by taking the entireimage or performing an image cropping operation to select a portion ofsaid image.

In various embodiments, said second simulated focal length is equal toor larger than the second focal length. In some embodiments, said firstdigital zoom is accomplished by taking the entire image or performing animage cropping operation to select a portion of said image.

In some embodiments, the digital zoom during said second period of timebegins with the second simulated focal length and ends with a thirdsimulated focal length.

In some such embodiments, said second simulated focal length is equal toor larger than the second focal length. In some such embodiments, saidsecond digital zoom is accomplished by taking the entire images orperforming image cropping operations to select a portion of said images

In various embodiments, said third simulated focal length is equal to orlarger than the second focal length. In some such embodiments, saidsecond digital zoom is accomplished by taking the entire image (equal tolens focal length) or performing an image cropping operation to select aportion of said image (simulating a larger focal length).

In some embodiments, the fourth simulated focal length is greater thanor equal to the first focal length.

In some embodiments, the fifth simulated focal length is greater than orequal to the second focal length.

In some embodiments, in which the fourth simulated focal length is lessthan the first simulated focal length, said first simulated focal lengthis less than the second simulated focal length, the second simulatedfocal length is less than the third simulated focal length and the thirdsimulated focal length is less than the fifth simulated focal length(for zooming in) and wherein the first focal length is less than thesecond focal length.

In some embodiments in which the fourth simulated focal length isgreater than the first simulated focal length, said first simulatedfocal length is greater than the second simulated focal length, thesecond simulated focal length is greater than the third simulated focallength and the third simulated focal length is greater than the fifthsimulated focal length (for zooming out) and wherein the first focallength is greater than the second focal length.

In various embodiments, the rate of change of the In (natural log) ofthe simulated focal length during said first, second, third and fourthtime periods is the same and is constant, e.g., resulting in continuouszoom at a fixed rate.

In FIG. 25, elements are identified using the first two digits “25” butthe same last two digits used in FIG. 22 to identify the stepimplemented by the module shown in FIG. 25. For example, module 2506 isconfigured to perform the operations of step 2206, while module 2508 isconfigured to perform the operations corresponding to step 2208. Sincethe steps and operations implemented by the modules shown in FIG. 25have already been described in the context of FIG. 22 they will not bedescribed further. While the FIG. 25 embodiment shows different modulesbeing used for different time periods it should be appreciated that inat least some embodiments an individual module is used and controlled toperform in accordance with the operations to be performed during theparticular time period in which the module is being operated. Forexample, in some embodiments an image capture module is used and/orcontrolled to perform the operations corresponding to the various imagecapture modules of FIG. 25 in accordance with what particular timeperiod the image capture module is being operated in. Similarly imagecombining may be performed by a module which operates in a timedependent manner so that it implements the image combining operations tobe performed for a particular time period. Similarly, a single imagedata exclusion module may be used with the image data exclusion moduleperforming image data exclusion based on which time period particularcaptured image data corresponds to. Thus, while separate modules may beused as shown in FIG. 25 for the different time periods in someembodiments modules which operate or are controlled to operate as afunction of time may be used to perform the functions of the particulartype of module for multiple different time periods.

Is should be noted that in at least some embodiments the modules of FIG.25 are implemented fully in hardware, e.g., as circuits with anindividual circuit performing the function of the corresponding module.However, in other embodiments modules are implemented in software or acombination of software and hardware.

The method of FIG. 22 and assembly of modules shown in FIG. 25 can beused in a camera device including optical modules shown in any one ofthe other figures which can be used in the manner described in FIG. 22.As should be appreciated the method shown in FIG. 22 is flexible andwell suited for a wide range of lens and filter configurations.

In various embodiments images captured by optical chains are stored,processed, transmitted to another device for processing and/ordisplayed, e.g., on a screen of the camera device. Similarly, acomposite image generated by a camera device or another device fromcaptured images is stored, processed, transmitted to another deviceand/or displayed, e.g., on a screen or other display of the cameradevice.

In various embodiments, the camera device provides a depth map on thedevice to the user at the time of taking a shot. An object that is at aslant angle or a large group of people may, and sometimes does, fallinto several “colors” of depth buckets, and the user may want the entireobject or the large group of people to be in-focus and want thebackground to be blurred. In some embodiments, such as the examples ofFIGS. 22-25, the user is given the option to pick multiple depth bucketsto be in-focus. The user can select his or her preference based on thedepth map, e.g., clicking all of the depth colors that correspond topeople in the group.

In some embodiments, the final chosen in-focus range is made continuousregardless of user selection, e.g., the camera automatically includescolors within the limits of the selected user range to be included aspart of the in-focus range.

FIG. 26, comprising the combination of FIG. 26A and FIG. 26B, is aflowchart 2600 of an exemplary method of using multiple optical chainsincluding at least a first group of optical chains and a second group ofoptical chains, in accordance with an exemplary embodiment. In someembodiments, the exemplary method of flowchart 2600 is implemented by acamera device, e.g., camera device 2700 of FIG. 27. In some otherembodiments, some portions of the method of flowchart 2600 areimplemented by a camera device, e.g., camera device 2700 of FIG. 27, andother portions of flowchart 2600, e.g., image processing steps, areimplemented by another device, e.g., computer system 1400 of FIG. 14. Insome such embodiments, captured images are output to another device,e.g., computer system 1400 of FIG. 14, in which image processing stepsare performed. In various embodiments, optical chains in the first groupof optical chains have a first fixed focal length, and optical chains inthe second group of optical chains have a second fixed focal lengthwhich is smaller than the first fixed focal length.

The exemplary method of flowchart 2600 will be described for anexemplary embodiment in which a camera device implements the steps offlowchart 2600. Steps indicated by dotted lines, in flowchart 2600 areoptional steps and may be omitted or bypassed in some embodiments.Operation starts in step 2602 in which the camera device is powered onand initialized. Operation proceeds from step 2602 to step 2604. In step2604 the camera device captures images using a third group of opticalchains with a third fixed focal length, said third fixed focal lengthbeing smaller than a first fixed focal length and a second fixed focallength. Operation proceeds from step 2604 to step 2606. In step 2606 thecamera device uses images captured by the third group of optical chainsto generate a depth map. In some such embodiments, the depth map is usedin combining images captured by different optical chains in said firstgroup of optical chains or the second group of optical chains. Operationproceeds from step 2606 to step 2608, and, in some embodiments, to step2610.

In step 2608, the camera device captures images using a first group ofoptical chains, e.g., with a first fixed focal length, during a firsttime period. In step 2610, the camera device captures images using thethird group of optical chains during the first time period. Operationproceeds from step 2608 and 2610 to step 2612. In step 2612 the cameradevice generates a first composite image from images captured during thefirst period of time using said first group of optical chains. In someembodiments, step 2612 includes step 2614, in which the camera deviceuses images captured during the first time period using the third groupof optical chains in generating the first composite image, e.g., croppedportions of images captured during the first time period using the thirdgroup of optical chains are used in generating the first compositeimage. Operation proceeds from step 2612 to step 2616.

In step 2616, the camera device stores, displays, and/or outputs thefirst composite image to another device, e.g., computer system 1400 ofFIG. 14 or a storage device. Operation proceeds from step 2616 to step2618. In step 2618 the camera device switches from using said firstgroup of optical chains, e.g., with a first fixed focal length, to asecond group of optical chains, e.g., with a second fixed focal length,in response to a user zoom control input, e.g., a zoom out user controlinput used to indicate a user intent to capture an image correspondingto a larger scene area than a scene area being captured when the userzoom out is initiated. Thus, in step 2618 the camera device switchesfrom a larger focal length group to a smaller focal length group as partof zoom out. Operation proceeds from step 2618 to step 2620, and in someembodiments, to step 2622.

In step 2620 the camera device captures images using a second group ofoptical chains during a second time period, optical chains in saidsecond group have a different focal length than optical chains in saidfirst group. In stop 2622 the camera device captures images using thethird group of optical chains during the second time period. Operationproceeds from step 2620 and step 2622, via connecting node A 2624, tostep 2626.

In step 2626 the camera device generates a second composite image fromimages captured during the second time period using said second group ofoptical chains. In some embodiments, step 2626 includes step 2628 inwhich the camera device uses images captured during the second timeperiod using the third group of optical chains in generating the secondcomposite image, e.g., cropped portion of images captured during thesecond time period using the third group of optical chains in generatingthe second composite image. Operation proceeds from step 2626 to step2630, in which the camera device stores, displays, and/or outputs thesecond composite image to another device. Operation proceeds from step2630 to step 2632.

In step 2632, the camera device determines if the user selected zoomsetting corresponds to a zoom level between a first zoom levelcorresponding to the first focal length and a second zoom levelcorresponding to the second focal length and controls operation as afunction of the determination. If the camera device determines that theuser selected zoom setting corresponds to a zoom level between a firstzoom level corresponding to the first focal length and a second zoomlevel corresponding to the second focal length, then operation proceedsfrom step 2632 to step 2634; otherwise, operation proceeds from step2632 to step 2640.

Returning to step 2634, in step 2634 the camera device performs adigital zoom operation to support a user selected zoom settingcorresponding to a focal length between the first and second focallengths. In various embodiments, step 2634 includes step 2636, in whichthe camera device performs a crop and interpolation operation togenerate a third composite image. In some embodiments, a digital zoom isimplemented by the camera device as a method of changing the apparentview of a digital photographic or video image being generated. In someembodiments, a digital zoom is accomplished by cropping an image down toa centered area with the same aspect ratio as the original, and in somebut not necessarily all embodiments, also interpolating the result backup to the pixel dimensions of the original. In some such embodiments,digital zoom is accomplished computationally. Operation proceeds fromstep 2634 to step 2638, in which the camera device stores, displays,and/or outputs the third composite image to another device. Operationproceeds from step 2638 to step 2640.

In step 2640 the camera device switches from using the second group ofoptical chains to said first group of optical chains in response to auser zoom in control input used to indicate a user intent to capture animage corresponding to a smaller area using a higher zoom setting thanwas being used at the time the user zoom in control input was activated.Operation proceeds from step 2640, via connecting node B 2642 to step2604.

In some embodiments, the first group of optical chains includes at leastfour different optical chains corresponding to a first focal length. Insome such embodiments, the four optical chains capture different portionof a scene area. In some such embodiments, different portions of thescene area at least partially overlap. In various embodiments, themultiple optical chains include a third group of optical chains have athird fixed focal length which is different from the first fixed focallength of the optical chains in the first group and the second fixedfocal length of the optical chains in the second group. In some suchembodiments, the third group of optical chains includes one or moreoptical chains, and the first and second groups of optical chainsinclude multiple optical chains. In some such embodiments, the number ofoptical chains in the first group is at least 3. In some suchembodiments, the number of optical chains in the second group is atleast 3. In some such embodiments, the number of optical chains in thethird group is at least 2. In some embodiments, images captured byoptical chins in at least two groups of optical chains are used togenerate a composite image.

FIG. 27 is a drawing of an exemplary camera device 2700, e.g., a cellphone including a camera, in accordance with an exemplary embodiment. Invarious embodiments, exemplary camera device 2700 implements a method inaccordance with flowchart 2600.

The camera device 2700, in some embodiments, is a portable device, e.g.,a cell phone or tablet including a camera assembly. In otherembodiments, it is fixed device such as a wall mounted camera. In stillother embodiments, the camera device is a portable camera.

The exemplary camera device 2700 includes a display device 2702, aninput device 2706, memory 2708, a processor 2710, a transceiverinterface 2714, e.g., a cellular interface, a WIFI interface, and/or aUSB interface, an I/O interface 2712, and a bus 2716 which are mountedin a housing represented by the rectangular box touched by the lineleading to reference number 2700. The input device 2706 may be, and insome embodiments is, e.g., keypad, touch screen, or similar device thatmay be used for inputting information, data and/or instructions. Thedisplay device 2702 may be, and in some embodiments is, a touch screen,used to display images, video, information regarding the configurationof the camera device, and/or status of data processing being performedon the camera device. In the case where the display device 2702 is atouch screen, the display device 2702 serves as an additional inputdevice and/or as an alternative to the separate input device, e.g.,buttons, 2706. The I/O interface 2712 couples the display 2702 and inputdevice 2706 to the bus 2716 and interfaces between the display 2702,input device 2706 and the other elements of the camera which cancommunicate and interact via the bus 2716. In addition to being coupledto the I/O interface 2712, the bus 2716 is coupled to the memory 2708,processor 2710, a transceiver interface 2714, and a plurality of opticalchain modules 2730, e.g., N optical chain modules. The plurality ofoptical chains module 2730 includes a plurality of groups of opticalchains modules (first group of optical chains 2732, second group ofoptical chains 2734, . . . , Mth group of optical chains 2736.) In oneexemplary embodiment, the first group of optical chains 2732 have afirst fixed focal length=f1, the second group of optical chains 2734have a second fixed focal length=f2, and the Mth group of optical chains2736, where M=3, have a third fixed focal length=f3. In some suchembodiments, f1>f2>f3.

The first group of optical chains 2732 includes a plurality of opticalchains (group 1 optical chain 1 2738, . . . , group 1 optical chain N12740). The second group of optical chains 2734 includes a plurality ofoptical chains (group 2 optical chain 1 2742, . . . , group 2 opticalchain N2 2743). The Mth group of optical chains 2736 includes one ormore optical chains (group M optical chain 1 2746, . . . , group Moptical chain N3 2748).

In some embodiments the number of optical chains, N, in the cameradevice 2700 is an integer greater than 4, e.g., 5, 7, 8, 9, 10 or alarger value depending on the particular embodiment. Images captured byindividual optical chain modules in the plurality of optical chainmodules 2730 can be stored in memory 2708, e.g., as part of thedata/information 2720 and processed by processor 2710 and/or processor2754, e.g., to generate one or more composite images. Composite imagesmay also be stored in memory 2708, e.g., as part of data/information2720. In various embodiments, images from one or more selected groups ofoptical chains are used to generate a composite image, e.g., with theparticular one or more selected groups which are used being a functionof a zoom setting, e.g. a user selected zoom setting.

In some embodiments, multiple captured images and/or composite imagesmay be processed to form video, e.g., a series of images correspondingto a period of time. Transceiver interface 2714 couples the internalcomponents of the camera device 2700 to an external network, e.g., theInternet, and/or one or more other devices e.g., memory or stand alonecomputer. Via interface 2714 the camera device 2700 can and does outputdata, e.g., captured images, generated composite images, and/orgenerated video. The output may be to a network or to another externaldevice for processing, storage and/or to be shared. The captured imagedata, generated composite images and/or video can be provided as inputdata to another device for further processing and/or sent for storage,e.g., in external memory, an external device or in a network.

The transceiver interface 2714 of the camera device 2700 may be, and insome instances is, coupled to a computer so that image data may beprocessed on the external computer. In some embodiments the externalcomputer has a higher computational processing capability than thecamera device 2700 which allows for more computationally complex imageprocessing of the image data outputted to occur on the externalcomputer. The transceiver interface 2714 also allows data, informationand instructions to be supplied to the camera device 2700 from one ormore networks and/or other external devices such as a computer or memoryfor storage and/or processing on the camera device 2700. For example,background images may be supplied to the camera device to be combined bythe camera processor, e.g., processor 2710 and/or image processor 2754with one or more images captured by the camera device 2700. Instructionsand/or data updates can be loaded onto the camera via interface 2714 andstored in memory 2708.

The camera device 2700 may include, and in some embodiments doesinclude, an autofocus controller 2727 and/or autofocus drive assembly2729. The autofocus controller 2727 is present in at least someautofocus embodiments but would be omitted in fixed focus embodiments.

Camera device 2700 further includes a user zoom control 2752, and acontrol device 2750 coupled to the bus 2716. In some embodiments, thecontrol device 2750 includes a processor 2750. In some embodiments, thecamera device 2700 includes an image processor 2754, e.g., a dedicatedcustomized image processor. In various embodiments, the image processor2754 and the control device 2750 are included as part of an integratedcircuit 2760. In some embodiments, processor 2710, image processor 2754and control processor 2756 are included as part of an integratedcircuit.

One or more or all of the processors (2710, 2756, 2754) controlsoperation of the camera device 2700 to control the elements of thecamera device 2700 to implement the steps of the methods describedherein. One or more of the processors may be a dedicated processor thatis preconfigured to implement the method or portions of the method. Forexample, in one embodiment, image processor 2754 performs imageprocessing operations including, e.g. combining captured images frommultiple optical chains, cropping images, etc., processor 2756 performsvarious control operation relating to the camera, e.g., switchingbetween groups of optical chains, etc., and processor 2710 performsvarious other function such as receiving and processing user input anddisplaying output on the display, etc. However, in many embodiments theprocessor or processors (2710, 2754, and/or 2756) operate underdirection of software modules and/or routines stored in the memory 2708which include instructions that, when executed, cause the one or moreprocessors to control the camera device 2700 to implement one, more orall of the methods described herein. Memory 2708 includes an assembly ofmodules 2718 wherein one or more modules include one or more softwareroutines, e.g., machine executable instructions, for implementing theimage capture and/or image data processing methods of the presentinvention. Individual steps and/or lines of code in the modules of 2718when executed by the processor (2710, 2754, 2756) control the processor(2710, 2754, 2756) to perform steps of the method of the invention. Whenexecuted by processor (2710, 2754, 2756) the data processing modulesincluded in assembly of modules 2718 cause at least some data to beprocessed by the processor (2710, 2754, 2756) in accordance with themethod of the present invention. The resulting data and information(e.g., captured images of a scene, combined images of a scene, etc.) arestored in data memory 2720 for future use, additional processing, and/oroutput, e.g., to display device 2702 for display or to another devicefor transmission, processing and/or display. The memory 2708 includesdifferent types of memory for example, Random Access Memory (RAM) inwhich the assembly of modules 2718 and data/information 2720 may be, andin some embodiments are stored for future use. Read only Memory (ROM) inwhich the assembly of modules 2718 may be stored for power failures.Non-volatile memory such as flash memory for storage of data,information and instructions may also be used to implement memory 2708.Memory cards may be added to the device to provide additional memory forstoring data (e.g., images and video) and/or instructions such asprogramming. Accordingly, memory 2708 may be implemented using any of awide variety of non-transitory computer or machine readable mediumswhich serve as storage devices.

In various embodiments, the camera device 2700 includes multiple opticalchains, and the multiple optical chains including at least a first groupof optical chains 2732 and a second group of optical chains 2734, andoptical chains in said first and second groups of optical chains (2732,2734) have different focal lengths.

Control device 2750 is configured to control said multiple opticalchains, and control includes: controlling optical chains in said firstgroup of optical chains 2732 during a first time period to captureimages; controlling optical chains in said second group of opticalchains 2734 during a second time period to capture images.

In various embodiments, image processor 2754 is configured to: generatea first composite image from images captured during said first timeperiod using said first group of optical chains; and generate a secondcomposite image from images captured during said second time periodusing said second group of optical chains.

In various embodiments, multiple optical chains, included in theplurality of optical chain modules 2730, control device 2730 andprocessor 2754 are included in camera device 2700.

In some embodiments, multiple optical chains included in the pluralityof optical chain modules 2730, and control device 2750 are included incamera device 2700 and said image processor is included in a deviceexternal to said camera device 2700, e.g., the image processor isprocessor 1410 of computer system 1400 of FIG. 14.

In some embodiments, control device 2750 includes a first processor,e.g., processor 2756 in camera device 2700 and the image processor is asecond processor, e.g., processor 2754 in camera device 2700 orprocessor 1410 in computer system 1400. For example, different chips areused within camera device 2700 or images are captured by camera device2700 under the control of processor 2756 in control device 2750 and theimages are processed separately on a PC, e.g., computer system 1400using processor 1410.

In some embodiments, control device 2750 and image processor 2754 areimplemented on a single integrated circuit 2760.

In various embodiments, optical chains (2738, . . . , 2740) in the firstgroup of optical chains 2732 have a first fixed focal length, e.g., f1;and optical chains (2742, . . . , 2744) in said second group of opticalchains 2734 have a second fixed focal length, e.g., f2, which is smallerthan said first fixed focal length, e.g., f2<f1.

In various embodiments, control device 2750 is further configured toswitch from using said first group of optical chains 2732 to said secondgroup of optical chains 2734 in response to a user zoom control settingfrom the user zoom control 2754.

In various embodiments, user zoom control input, e.g., received by theuser zoom control 2752, allows a user to zoom out and thereby select azoom setting used to capture or generate an image corresponding to alarger scene area than a scene area being captured when the user zoomout was initiated. Thus a switch from a large focal length group to asmaller focal length group is performed as a part of zoom out operation.

Zoom control module, e.g., a module included in assembly of modules 2718and loaded into processor 2756 or included as part of processor 2756,e.g., as circuitry, or a module included in assembly of modules 2718 andloaded into processor 2754 or included as part of processor 2754, e.g.,as circuitry controls switching from using images captured by saidsecond group of optical chains to said first group of optical chains inresponse to a user initiated zoom in operation used to indicate a userintent to capture an image corresponding to a smaller scene area using ahigher zoom setting than was being used at the time the user zoom incontrol input was activated.

In various embodiments, image processor 2754 is configured to perform adigital zoom operation to support a user selected zoom settingcorresponding to a focal length between said first and second focallengths.

In various embodiments, image processor 2754 is configured to perform adigital zoom operation, in response to a user selecting a zoom settingwhich corresponds to the zoom level which is between a first zoom levelcorresponding to said first focal length and a second zoom levelcorresponding to said second focal length. In various embodiments, imageprocessor 2754 is configured to perform a crop and interpolationoperation as part of said digital zoom operation.

In some embodiments, the first group of optical chains 2732 includes atleast four different optical chains corresponding to a first focallength. In some such embodiments, the four optical chains in said firstgroup 2732 capture different portions of a scene area. In some suchembodiments, said different portions of the scene area at leastpartially overlap.

In various embodiments, the multiple optical chains 2730 furtherincludes a third group of optical chains 2736 having a third fixed focallength which is different from the first fixed focal length of theoptical chains in the first group 2732 and the second fixed focal lengthof the optical chains in the second group 2734, said third group 2736 ofoptical chains including one or more optical chains, said first andsecond groups of optical chains each including multiple optical chains.

In some embodiments, the number of optical chains in said first group2732 of optical chains is at least 3, e.g., N1 is at least 3. In somesuch embodiments, the number of optical chains in said second group 2734is at least three, e.g., N2 is at least 3. In some such embodiments, thenumber of optical chains in said third group 2736 is at least two, e.g.,N3 is at least 2.

In various embodiments, images captured by optical chains from at leasttwo groups of optical chains of different focal lengths are used togenerate a composite image, e.g., images captured by the first and thirdgroups of optical chains are used to generate a composite image orimages captured by the second and third groups are used to generate acomposite image.

In various embodiments, the third focal length, e.g., corresponding tothe third group, is smaller than the first and second focal lengths,e.g., corresponding to the first and second groups respectively, andimage processor 2754 is further configured to: use images captured bysaid third group of optical chains to generate a depth map used incombining images captured by different optical chains in said firstgroup of optical chains or said second group of optical chains.

In some embodiments, an exemplary image capture system includes cameradevice 2700 of FIG. 27. In some other embodiments, an exemplary imagecapture system includes camera device 2700 and computer system 1400,e.g., a PC, of FIG. 14.

FIG. 28 is a diagram 1200 showing how the 17 optical chains, e.g.,camera modules, of a camera implemented in accordance with an exemplaryembodiment can be arranged within the body of the camera device such asthe camera device of FIG. 2, FIG. 27 or any of the other camera devicesor cameras described in the present application. The 17 optical chainscorrespond to three different groups when considered from theperspective of the focal length of the optical chains. The seven opticalchains 1202, 1206, 1210, 1212, 1216 1220, 1222 with the largest lensesand/or outer opens, and have the largest focal lengths have focal lengthF1. F1 in some embodiments is equivalent to a 140 mm lens used on a 35mm conventional film camera. Thus the 140 mm focal length of the opticalchain including a sensor is what is sometimes referred to as a 35 mmequivalent focal length and it is a measure that indicates the angle ofview of the particular combination of a camera lens, lenses or otheroptical elements of optical chain and the sensor size of the sensor inthe optical chain. The five camera modules 1204, 1208, 1214, 1218, 1224with the medium diameter lenses and medium supported focal lengthscorrespond to a focal length F2 which is smaller than F1. In someembodiments F2 is a focal length equivalent to a 70 mm lens of a 35 mmconventional film camera. The five optical chains 1226, 1228, 1230, 1232and 1234 having the smallest diameter outer lenses and smallest focallengths and are implemented using optical chains which have a focallength F3. In some embodiments F3 is equivalent to a 35 mm focal lengthlens of a conventional 35 mm film camera. Each of the optical chainsshown in FIG. 28 include at least one lens and a sensor. Some or all ofthe optical chain modules may include a mirror or other lightredirection device in addition to the mirror and lens.

In some embodiments the focal lengths of the optical chain modules shownin FIG. 28 are fixed. In other embodiments, the optical chain modulescan change their focal length, e.g., from one discrete focal length toanother, e.g., by moving one or more of the elements, e.g., lenses ormirror of the optical chain module. The focal lengths of some or all ofthe optical chain modules, in one or more of the groups, may be changedfrom one discrete focal length setting to another discrete focal lengthsetting in some embodiments.

The image capture area of different optical chain modules correspondingto different focal length groups can be appreciated from FIG. 29. FIG.29 includes is a diagram 3200 which shows scene areas captured bydifferent optical chain modules of the camera device 1200 when they areoperated in parallel to capture images to be used to generate acomposite image, e.g., at a user controlled zoom setting.

Scene areas 3302, 3310, 3320, 3312 are captured by camera modules havingfocal lengths F1, e.g., the longest focal lengths of the camera device1200 and which thus capture the smallest scene area. For example scenearea 3302 can be captured by camera module 1202, scene area 3310 can becaptured by camera module 1210, scene area 3320 can be captured bycamera module 1220 and scene area 3312 can be captured by camera module1212 operating in parallel, e.g., to capture images at the same time.While not shown in FIG. 29 to avoid confusion by the presence of anexcessive number of boxes, in some embodiments a fifth F1 camera module1216 is used to capture an F1 Scene area at the center of the sceneshown in FIG. 29.

One or more camera modules corresponding to the other focal lengthgroups, e.g., F2 and F3, can, and sometimes are, also be operated tocapture images while the images of areas 3302, 3310, 3320, 3312 arecaptured. For example, F2 module 1204 is sometimes operated to capturescene area 3206, module 1208 is operated to capture scene area 3208,module 1214 is operated to capture scene area 3214 and module 1218 isoperated to capture scene area 3312. In addition, F2 module 1224 isoperated in some embodiments to capture scene area 3216.

One or more F3 camera modules capture images of the scene area 3210. forexample camera modules 1234, 1230 are operated in some embodiments tocapture images of scene area 3210. While images can and sometimes arecaptured in parallel by two or more groups of camera modules withdifferent focal lengths, e.g., two or 3 of the groups explained withreference to FIG. 28 and/or FIG. 29, which captured images are used togenerate a composite image are normally controlled based on a user zoomsetting. Thus, in some embodiments, depending on the user zoom settingdifferent optical chains and/or groups of optical chains are used tocapture images which will be used to generate the composite imagecorresponding to the user selected zoom setting.

Images which are not to be used to generate a composite image, even ifcaptured by a camera module, are not stored in the memory of the cameradevice for purposed of composite image generation in some embodiments tosave memory. Thus, in at least some embodiments, based on the user zoomsetting some optical chain modules are not used and/or captured imagesare exclude from use in generating the composite image. For example, insome embodiments when the zoom setting indicates a user desire tocapture the scene area 3216, the F2 camera module used to capture thescene area 3216 may be used in combination with the F1 camera modulesused to capture scene areas 3302, 3320, 3310 and 3312 but the F3 cameramodule(s) used to capture scene area 3210 may be left unused. However,when a user selected zoom setting indicates a desire to capture thescene area 3210 the one or more F3 camera modules which capture scenearea 3210 are used in combination with the F2 camera modules whichcaptures scene areas 3206, 3208, 3212, 3214. In this manner a scene areacorresponding to a user selected zoom setting may be captured by cameramodules corresponding to multiple different focal lengths. Use of theimage areas captured by the shorter focal lengths facilitates combing ofimage captured by the longer length focal length modules.

FIG. 30 illustrates an exemplary scene 3300 that may be captured usingthe F3 and F2 camera modules in combination as discussed with regard toFIG. 29 to generate a composite image corresponding to a F3 userselected zoom setting. The FIG. 29 example assumes the outputs of the F1camera modules are not used in generating the composite image for the F3zoom setting but it should be appreciated that they could be, and insome embodiments are, used as well to generate the composite image. Anumber followed by a prime is used to indicate the scene portioncorresponding to the scene area with the corresponding number of FIG.29. For example, exemplary scene portion 3206′ corresponds to secondscene area 3206 while scene portion 3208′ corresponds to scene area3208.

It should be appreciated from the discussion of FIGS. 29 and 30 thatdifferent groups and/or numbers of camera modules may, and sometimesare, used at different times to generate composite images correspondingto user zoom settings.

Numerous additional variations and combinations are possible whileremaining within the scope of the invention. Cameras implemented in someembodiments have optical chains which do not extend out beyond the frontof the camera during use and which are implemented as portable handheldcameras or devices including cameras. Such devices may and in someembodiments do have a relatively flat front with the outermost lens orclear, e.g., (flat glass or plastic) optical chain covering used tocover the aperture at the front of an optical chain being fixed.However, in other embodiments lenses and/or other elements of an opticalchain may, and sometimes do, extend beyond the face of the cameradevice.

In various embodiments the camera devices are implemented as digitalcameras, video cameras, notebook computers, personal data assistants(PDAs), or other portable devices including receiver/transmittercircuits and logic and/or routines, for implementing the methods of thepresent invention and/or for transiting captured images or generatedcomposite images to other devices for storage or display.

The techniques of the present invention may be implemented usingsoftware, hardware and/or a combination of software and hardware. Thepresent invention is directed to apparatus, e.g., dedicated cameradevices, cell phones, and/or other devices which include one or morecameras or camera modules. It is also directed to methods, e.g., methodof controlling and/or operating cameras, devices including a camera,camera modules, etc. in accordance with the present invention. Thepresent invention is also directed to machine readable medium, e.g.,ROM, RAM, CDs, hard discs, etc., which include machine readableinstructions for controlling a machine to implement one or more steps inaccordance with the present invention.

In various embodiments devices described herein are implemented usingone or more modules to perform the steps corresponding to one or moremethods of the present invention, for example, control of image captureand/or combining of images. Thus, in some embodiments various featuresof the present invention are implemented using modules. Such modules maybe implemented using software, hardware or a combination of software andhardware. In the case of hardware implementations embodimentsimplemented in hardware may use circuits as part of or all of a module.Alternatively, modules may be implemented in hardware as a combinationof one or more circuits and optical elements such as lenses and/or otherhardware elements. Thus in at least some embodiments one or moremodules, and sometimes all modules, are implemented completely inhardware. Many of the above described methods or method steps can beimplemented using machine executable instructions, such as software,included in a machine readable medium such as a memory device, e.g.,RAM, floppy disk, etc. to control a machine, e.g., a camera device orgeneral purpose computer with or without additional hardware, toimplement all or portions of the above described methods, e.g., in oneor more nodes. Accordingly, among other things, the present invention isdirected to a machine-readable medium including machine executableinstructions for causing or controlling a machine, e.g., processor andassociated hardware, to perform e.g., one or more, or all of the stepsof the above-described method(s).

While described in the context of an cameras, at least some of themethods and apparatus of the present invention, are applicable to a widerange of image captures systems including tablet and cell phone deviceswhich support or provide image capture functionality.

Images captured by the camera devices described herein may be real worldimages useful for documenting conditions on a construction site, at anaccident and/or for preserving personal information whether beinformation about the condition of a house or vehicle. Captured imagesand/or composite images maybe and sometimes are displayed on the cameradevice or sent to a printer for printing as a photo or permanentdocument which can be maintained in a file as part of a personal orbusiness record.

Numerous additional variations on the methods and apparatus of thepresent invention described above will be apparent to those skilled inthe art in view of the above description of the invention. Suchvariations are to be considered within the scope of the invention. Invarious embodiments the camera devices are implemented as digitalcameras, video cameras, notebook computers, personal data assistants(PDAs), or other portable devices including receiver/transmittercircuits and logic and/or routines, for implementing the methods of thepresent invention and/or for transiting captured images or generatedcomposite images to other devices for storage or display.

Numerous additional embodiments are possible while staying within thescope of the above discussed features.

What is claimed:
 1. A method of using multiple optical chains of acamera device, said multiple optical chains including at least a firstgroup of optical chains and a second group of optical chains, the methodcomprising: capturing images using said second group of optical chainsduring a first time period, said second group of optical chainsincluding multiple optical chains having a second focal length which isfixed during said first time period, different optical chains in thesecond group of optical chains capturing different portions of a scenearea; checking a user selected zoom setting to determine if a zoom levelindicated by the user selection zoom setting corresponds to a zoom levelbetween a first zoom level corresponding to a first focal length and asecond zoom level corresponding to the second focal length; andperforming a digital zoom operation when the zoom level indicated by theuser selected zoom setting corresponds to a zoom level between the firstzoom level corresponding to the first focal length and the second zoomlevel corresponding to the second focal length; and switching from thesecond group of optical chains to another group of optical chains whenthe zoom level indicated by the user selected zoom setting correspondsto a zoom level that is not between the first zoom level correspondingto the first focal length and the second zoom level corresponding to thesecond focal length.
 2. The method of claim 1, wherein said second groupof optical chains includes multiple fixed focal length optical chainshaving the second focal length.
 3. The method of claim 2, wherein saidfirst group of optical chains includes multiple optical chains having afirst fixed focal length which is different from the second focallength.
 4. The method of claim 3, further comprising: generating a depthmap from images captured by multiple different optical chains in saidcamera device.
 5. The method of claim 4, further comprising: displayingthe depth map to a user of the camera device.
 6. The method of claim 5,further comprising: providing the user the opportunity to selectmultiple depths to be in focus.
 7. The method of claim 4, wherein saidcamera device further includes a third group of optical chains having ashorter focal length then said first or second focal length; and whereingenerating a depth map from images captured by multiple differentoptical chains in said camera device includes: using images captured bythe third group of optical chains to generate said depth map.
 8. Themethod of claim 3, further comprising: generating a first compositeimage from images captured using the different optical chains in saidsecond group of optical chains during said first time period.
 9. Themethod of claim 2, wherein the camera device further includes a thirdgroup of optical chains, the method further comprising: generating adepth map from images captured by said third group of optical chains.10. The method of claim 9, further comprising: displaying the depth mapto a user of the camera device.
 11. The method of claim 9, wherein thethird group of optical chains include optical chains having a smallerfocal length than optical chains in either said first group of opticalchains or said second group of optical chains.
 12. The method of claim11, wherein said camera device is a portable handheld camera thatincludes said first, second and third groups of optical chains.
 13. Themethod of claim 2, wherein said camera device is a portable handheldcamera that includes said first group of optical chains, said secondgroup of optical chains and a third group of optical chains.